Visual detection of guava maturity and method and device for controlling picking

Abstract

This invention provides a method and apparatus for visual detection and harvesting control of guava maturity, relating to the field of visual detection technology. It acquires multi-view time-series images from a drone and extracts individual guava image blocks and depth information to determine spatial location. A feature encoder extracts feature vectors and performs clustering to determine maturity levels. Then, an optimization problem is constructed to plan the harvesting path, aiming to maximize harvesting revenue per unit time. Finally, the drone and robotic arm are controlled for flexible harvesting. This method can achieve automatic identification of guava maturity, 3D positioning, revenue-driven target selection and path planning, and precise flexible harvesting even in the absence of numerous manual maturity labels. It forms an integrated autonomous harvesting system with a closed loop of perception, understanding, decision-making, and execution, effectively solving problems such as difficult feature extraction, high manual labeling costs, low planning efficiency, and fruit damage in greenhouse guava harvesting.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a method and apparatus for visual inspection of guava maturity and harvesting control. Background Technology

[0002] With the continuous development of modern agriculture and intelligent agricultural machinery technology, the use of unmanned aerial vehicles (UAVs) equipped with robotic arms for automated and intelligent fruit harvesting in greenhouse environments has become an important development trend. For cash crops such as guava with specific ripening cycles, users need to accurately identify the ripeness of guava and plan efficient harvesting routes to complete autonomous operations when carrying out harvesting tasks.

[0003] In terms of technical implementation, existing guava maturity recognition schemes typically rely on supervised deep learning models. These methods require the pre-collection of a large number of guava images and significant human resources for manual labeling of maturity levels. However, in actual greenhouse environments, guava trees are severely shaded, have complex lighting conditions, and cluttered backgrounds, resulting in poor generalization ability of supervised models. Furthermore, maturity is a continuous evolutionary process over time, and relying on human experience to forcibly assign discrete labels is extremely costly, greatly limiting the large-scale deployment of visual recognition systems.

[0004] Furthermore, existing path planning technologies for harvesting robots often focus solely on obstacle avoidance and reaching the target guava, neglecting the economic benefits of the task. In a single operation within a greenhouse, the number of guavas typically far exceeds the harvestable limit imposed by the drone's battery life. Simply performing pathfinding without considering the maturity value of guavas at different locations and the energy / time costs of the drone and robotic arm's movements will result in low returns per unit time for the harvesting system.

[0005] Moreover, guava is a fragile soft fruit crop, and drones inevitably experience slight swaying due to airflow or their own dynamics when operating in the air. Existing rigid position control methods cannot guarantee the precise alignment of the robotic arm's end effector under dynamic disturbances, and the gripping action is stiff, which can easily cause mechanical damage to the guava's skin or branches, seriously affecting the yield of harvested guava. Summary of the Invention

[0006] This invention provides a method and apparatus for visual detection of guava maturity and harvesting control, in order to overcome the deficiencies existing in related technologies.

[0007] This invention provides a method for visual detection and harvesting control of guava maturity, comprising: The system acquires time-series images from multiple perspectives captured by a camera mounted on a drone inside a greenhouse, and obtains guava fruit image blocks and depth information of the guava fruit image blocks within a single image in the time-series images. Based on the guava fruit image blocks and the depth information, the spatial position of each guava fruit in the single image is determined. The guava single fruit image patch is input into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder. The guava single fruit feature vector is clustered to obtain the category of each guava single fruit, and the maturity level corresponding to the category of each guava single fruit is determined. Based on the spatial location of each guava fruit and the maturity level, an optimization problem is constructed with the goal of maximizing the harvest maturity benefit per unit time. The optimization problem is then solved to obtain the target set of harvestable fruits and the corresponding harvesting access path. Based on the harvesting access path, the drone is sequentially controlled to fly to the spatial location of each target guava fruit in the target harvesting fruit set, and the robotic arm on the drone is controlled to perform flexible harvesting of each target guava fruit.

[0008] According to the present invention, a method for visual detection and harvesting control of guava maturity is provided, wherein the training steps of the feature encoder include: Each guava single fruit image sample in each batch was randomly enhanced twice to obtain two enhanced versions of each guava single fruit image sample; Two enhanced versions of each guava single fruit image sample are sequentially input into the feature encoder and the image projection head to obtain two projection vectors corresponding to each guava single fruit image sample output by the image projection head. Normalize the two projection vectors corresponding to each single guava fruit image sample to obtain two normalized results for each single guava fruit image sample, and calculate the batch average contrast loss based on the two normalized results for each single guava fruit image sample. The feature encoder and the image projector are jointly trained based on the batch average contrast loss.

[0009] According to the present invention, a method for visual detection and harvesting control of guava maturity is provided, wherein determining the spatial position of each guava fruit in a single image based on the guava fruit image block and the depth information includes: Based on the guava single fruit image block, determine the center pixel coordinates of the corresponding guava single fruit; Based on the camera's intrinsic parameter matrix, the coordinates of the center pixel, and the depth information corresponding to the center pixel of the guava fruit, the position of the guava fruit in the camera coordinate system is calculated. The spatial position of the guava fruit is calculated based on its position in the camera coordinate system.

[0010] According to the present invention, a method for visual detection and harvesting control of guava maturity is provided, wherein the step of sequentially controlling the drone to fly to the spatial location of each target guava in the target harvested fruit set based on the harvesting access path includes: The current image captured by the camera during the control process is acquired and the current image is identified. When the target guava fruit and the robotic arm are present in the current image, the target center pixel coordinates of the target guava fruit and the end effector pixel coordinates of the robotic arm are determined. Calculate the image plane error based on the target center pixel coordinates and the end effector pixel coordinates; Based on the image plane error, the end effector speed of the robotic arm is calculated, and the end effector speed is mapped to the joint speed of the robotic arm; The robotic arm is controlled based on the joint speed.

[0011] According to the present invention, a visual detection and harvesting control method for guava maturity is provided, wherein the optimization problem satisfies a time constraint, wherein the total path duration of the candidate harvesting path is less than or equal to the total available task time, and the total path duration is equal to the sum of the flight time between two adjacent candidate guava fruits in the candidate harvesting path and the sum of the harvesting time of each candidate guava fruit in the candidate harvesting path. The harvest maturity benefit is determined based on the sum of the maturity weights of each candidate guava fruit in the candidate harvesting path, and the maturity weights of each candidate guava fruit are obtained by mapping the maturity level of each candidate guava fruit.

[0012] According to the present invention, a method for visual detection and harvesting control of guava maturity is provided, wherein controlling the robotic arm mounted on the drone to perform flexible harvesting of each target guava fruit includes: Determine the desired pose and actual pose of the end effector of the robotic arm, and determine the desired contact force and actual contact force of the end effector of the robotic arm. Based on the desired pose and the actual pose of the end effector, the pose error is calculated, and based on the desired contact force and the actual contact force of the end effector, the force error is calculated. Based on the posture error and the force error, the impedance equation is applied to control the robotic arm mounted on the UAV to flexibly harvest each target guava fruit.

[0013] According to the present invention, a method for visual detection and harvesting control of guava maturity is provided, wherein obtaining a single guava image block within a single image in the time-series image includes: The single image is segmented to obtain multiple guava single-fruit masks; Based on the multiple guava single-fruit masks, the single image is cropped to obtain the guava single-fruit image block.

[0014] The present invention also provides a visual detection and harvesting control device for guava ripeness, comprising: The acquisition module is used to acquire time-series images from multiple perspectives captured by a camera mounted on a drone inside a greenhouse, and to acquire guava fruit image blocks and depth information of the guava fruit image blocks in a single image within the time-series images. Based on the guava fruit image blocks and the depth information, the spatial position of each guava fruit in the single image is determined. The maturity determination module is used to input the guava single fruit image block into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder, cluster the guava single fruit feature vector to obtain the category of each guava single fruit, and determine the maturity level corresponding to the category of each guava single fruit. The optimization problem construction module is used to construct an optimization problem based on the spatial location of each guava fruit and the maturity level, with the goal of maximizing the harvest maturity benefit per unit time, and solve the optimization problem to obtain the target set of harvestable fruits and the corresponding harvesting access path. The harvesting control module is used to control the drone to fly to the spatial position of each target guava fruit in the target fruit collection according to the harvesting access path, and to control the robotic arm on the drone to perform flexible harvesting of each target guava fruit.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the guava ripeness visual detection and harvesting control method as described above.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the guava ripeness visual detection and harvesting control method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the guava ripeness visual detection and harvesting control method as described above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a visual detection and harvesting control method and apparatus for guava maturity. It acquires multi-view time-series images from a drone and extracts image patches and depth information of individual guava fruits to determine their spatial location. A feature encoder extracts feature vectors and performs clustering to determine maturity levels. Then, an optimization problem is constructed to plan the harvesting path, aiming to maximize harvesting revenue per unit time. Finally, the drone and robotic arm are controlled for flexible harvesting. This method can achieve automatic identification of guava fruit maturity, 3D positioning, revenue-driven target selection and path planning, and precise flexible harvesting even in the absence of numerous manual maturity labels. It forms an integrated autonomous harvesting system with a closed loop of perception, understanding, decision-making, and execution, effectively solving problems such as difficult feature extraction, high manual labeling costs, low planning efficiency, and fruit damage in greenhouse guava harvesting, thus increasing harvesting revenue per unit time. Through flexible harvesting, this method can reduce damage to individual guava fruits during the harvesting process, increasing the yield of harvested guava. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the visual detection and harvesting control method for guava maturity provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of the guava maturity visual detection and harvesting control device provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 This is a flowchart illustrating a method for visual detection and harvesting control of guava maturity provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0025] S1, acquire time-series images from multiple perspectives captured by the camera mounted on the drone inside the greenhouse, and acquire guava single fruit image blocks and depth information of the guava single fruit image blocks in a single image in the time-series images. Based on the guava single fruit image blocks and the depth information, determine the spatial position of each guava single fruit in the single image.

[0026] S2, input the guava single fruit image block into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder, cluster the guava single fruit feature vector to obtain the category of each guava single fruit in the single image, and determine the maturity level corresponding to the category of each guava single fruit.

[0027] S3. Based on the spatial location of each guava fruit and the maturity level, an optimization problem is constructed with the goal of maximizing the harvest maturity benefit per unit time. The optimization problem is then solved to obtain the target set of harvestable fruits and the corresponding harvesting access path.

[0028] S4, based on the picking access path, the drone is sequentially controlled to fly to the spatial position of each target guava fruit in the target picking fruit set, and the robotic arm on the drone is controlled to perform flexible picking of each target guava fruit.

[0029] The guava maturity visual detection and picking control method provided in this embodiment of the invention is executed by a guava maturity visual detection and picking control device. This device can be configured in the main control equipment of a drone or a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0030] First, step S1 is executed to acquire time-series images from multiple perspectives captured by the camera mounted on the drone inside the greenhouse. Time-series images refer to a collection of multiple images captured by the camera mounted on the drone over multiple days and from different perspectives, which can reflect the continuous evolutionary trajectory of the same guava fruit from unripe to ripe in the time-series images.

[0031] Image segmentation algorithms, such as instance segmentation, can be used to identify single guava fruit image patches within a single image in a time-series image. A single guava fruit image patch is an image containing only a single guava fruit region, eliminating interference from background areas such as leaves and branches, and focusing on the individual fruit.

[0032] The drone can be equipped with a depth camera, which can determine the depth information of a single image in a time-series image. This depth information refers to the distance of each pixel in a single image relative to the camera within the greenhouse space. Therefore, after segmenting and obtaining individual guava fruit image patches, the depth information of each guava fruit image patch can be determined.

[0033] By utilizing individual guava fruit image patches and depth information, combined with the camera's intrinsic parameter matrix, the spatial position of each guava fruit in a single image is determined. This spatial information represents the three-dimensional position of the guava fruit in the world coordinate system. This step achieves stable visual scanning and three-dimensional perception of guava fruits within the greenhouse, providing an accurate target object and location basis for subsequent maturity assessment and harvesting planning.

[0034] Before acquiring time-series images from multiple perspectives captured by cameras mounted on a drone inside the greenhouse, the observable space inside the greenhouse can first be modeled as a three-dimensional region:

[0035] ;

[0036] in, This refers to the three-dimensional area inside the greenhouse where drones can fly. for The position vector of any point p in the vector. Let p be the coordinates of any point on the three coordinate axes. These are the lower and upper limits in the x-axis direction, respectively. These are the lower and upper limits in the y-axis direction, respectively. These represent the lower and upper limits in the z-axis direction, respectively.

[0037] When acquiring time-series images, the drone's onboard camera can operate along a sawtooth scanning trajectory. This sawtooth scanning trajectory can be represented as:

[0038] ;

[0039] in, For trajectory parameters, Trajectory parameters The corresponding drone location, Let x and y be the starting points of the sawtooth scan trajectory. Let x be the scanning speed of the UAV along the x-direction. Trajectory parameters The row number of the corresponding scan line. This represents the spacing between adjacent scan lines in the y-direction. The scanning altitude of the drone. This refers to the trajectory termination parameter for the sawtooth scan trajectory.

[0040] The sampling frequency of the camera mounted on the drone can be expressed as:

[0041] ;

[0042] in, This refers to the sampling frequency of the camera mounted on the drone, which is the distance between sampling points of two adjacent frames. Let x be the flight speed of the drone in the x-direction. This is the camera frame rate, measured in frames per second.

[0043] The image overlap rate and sampling frame rate of the camera on the drone need to meet the following conditions:

[0044] ;

[0045] in, The length of the projection of a single image onto the ground. This represents the image overlap rate.

[0046] The intrinsic parameter matrix of the camera mounted on the drone can be represented as:

[0047] ;

[0048] in, This is the intrinsic parameter matrix of the camera. These are the camera's focal lengths in the x and y directions, respectively, in pixels. These are the coordinates of the camera's principal point in the x and y directions of the image, respectively.

[0049] Next, step S2 is executed, whereby the guava single-fruit image patch is input into the feature encoder to obtain the guava single-fruit feature vector output by the feature encoder. The feature encoder is a deep neural network model trained using self-supervised contrastive learning. It can automatically learn and extract a high-dimensional vector representation reflecting the visual characteristics of guava from a large number of unlabeled guava single-fruit image samples, even in the absence of a large number of artificial maturity labels. The guava single-fruit feature vector is the high-dimensional representation extracted by the feature encoder. This feature vector not only integrates the color and texture features of the guava single fruit, but also reflects the changes in the fullness of the guava single fruit by combining depth information, and reflects the temporal evolution trend by combining time-series images.

[0050] Clustering is performed on the feature vectors of individual guava fruits to obtain the category of each guava fruit in a single image. The category of each guava fruit can be determined by several clusters with similar features obtained after unsupervised clustering of the extracted guava fruit feature vectors. Each cluster represents a set of fruits with similar visual features, indicating a category.

[0051] Here, clustering can be implemented using the K-means clustering algorithm, and the clustering objective can be expressed as:

[0052] ;

[0053] in, For the k-th cluster center, This represents the number of individual guava fruits within a single image. Let be the feature vector of the i-th guava fruit in a single image.

[0054] Each guava fruit can be represented as:

[0055] ;

[0056] in, Let be the cluster label of the i-th guava fruit in a single image, representing the category of the i-th guava fruit in a single image.

[0057] Furthermore, the maturity level corresponding to each guava fruit category is determined. The maturity level is determined by mapping the clustered guava fruit categories to maturity stages usable in actual production, such as unripe, color-changing, harvestable, and overripe, through minimal manual review or agricultural experience rules. By inputting guava fruit image patches into a feature encoder to extract feature vectors, and then performing unsupervised clustering and level mapping, the system can automatically learn and distinguish the features of guava fruits at different maturity stages without relying on large-scale manual labeling. This achieves unsupervised maturity stratification, reduces data labeling costs, and improves the automation level and accuracy of maturity identification.

[0058] Then, step S3 is executed, which uses the spatial location and maturity level of each guava fruit to construct an optimization problem with the goal of maximizing the harvest maturity benefit per unit time, and solves the optimization problem to obtain the target set of harvestable fruits and the corresponding harvesting access path.

[0059] The target guava set is the set of guava fruits to be picked in this task, selected from all identified and located guava fruits after solving an optimization problem. It takes into account the maturity value of the guava fruit, the flight and robotic arm movement costs caused by its spatial location, and the remaining time constraints such as battery or operation period.

[0060] The harvesting access path refers to the sequence or path planning result of harvesting each target guava fruit in the target harvesting fruit set. This step not only selects fruits that meet the ripeness requirements, but also takes into account harvesting efficiency and cost, transforming the system from simply harvesting as many as possible to harvesting just the right amount of fruit at the most cost-effective time, greatly improving the economic benefits of harvesting operations and the system utilization rate.

[0061] Finally, in step S4, the picking path is used in conjunction with visual servoing technology to plan the picking trajectory of the drone. Based on the picking trajectory, the drone is controlled to fly to the spatial position of each target guava fruit in the target fruit collection in sequence, and the robotic arm on the drone is controlled to gently pick each target guava fruit.

[0062] Flexible harvesting refers to the end effector of a robotic arm exhibiting a flexible, soft-hand characteristic when contacting and grasping individual guava fruits. It applies moderate gripping force upon contact, effectively absorbing rigid impact energy and preventing damage to the peel, thus avoiding mechanical damage to the fragile guava fruit. By controlling the drone's flight and the robotic arm's grasping actions based on a planned harvesting path and employing flexible harvesting technology, the robotic arm's end effector can accurately locate individual guava fruits in complex dynamic environments and provide appropriate force during grasping. This effectively adapts to the characteristics of soft-fruited crops like guava, improving harvesting success rates and reducing fruit loss.

[0063] The guava maturity visual detection and harvesting control method provided in this invention acquires multi-view time-series images from a drone and extracts image patches and depth information of individual guava fruits to determine their spatial location. It then uses a feature encoder to extract feature vectors and performs clustering to determine maturity levels. Furthermore, it constructs an optimization problem to plan the harvesting path, aiming to maximize harvesting revenue per unit time. Finally, it controls the drone and robotic arm for flexible harvesting. This method can achieve automatic identification of guava fruit maturity, 3D positioning, revenue-driven target selection and path planning, and precise flexible harvesting even in the absence of numerous manual maturity labels. It forms an integrated autonomous harvesting system with a closed loop of perception, understanding, decision-making, and execution, effectively solving problems such as difficult feature extraction, high manual labeling costs, low planning efficiency, and fruit damage in greenhouse guava harvesting, thus increasing harvesting revenue per unit time. Through flexible harvesting, this method can reduce damage to individual guava fruits during the harvesting process, increasing the yield of harvested guava.

[0064] Based on the above embodiments, the training steps of the feature encoder include: Each guava single fruit image sample in each batch was randomly enhanced twice to obtain two enhanced versions of each guava single fruit image sample; Two enhanced versions of each guava single fruit image sample are sequentially input into the feature encoder and the image projection head to obtain two projection vectors corresponding to each guava single fruit image sample output by the image projection head. Normalize the two projection vectors corresponding to each single guava fruit image sample to obtain two normalized results for each single guava fruit image sample, and calculate the batch average contrast loss based on the two normalized results for each single guava fruit image sample. The feature encoder and the image projector are jointly trained based on the batch average contrast loss.

[0065] Specifically, during the training of the feature encoder, each guava single-fruit image sample in each batch can be randomly augmented twice to obtain two augmented versions of each guava single-fruit image sample. The two random augmentations can include operations such as random cropping, color perturbation, and lighting simulation.

[0066] The two enhanced versions can be represented as follows:

[0067] ;

[0068] in, These are the image samples of the i-th guava fruit. Two enhanced versions, These are the data augmentation operators used for the two random augmentations, respectively.

[0069] The two enhanced versions of each guava single fruit image sample are sequentially input into the feature encoder and the image projection head, respectively, to obtain the two projection vectors corresponding to each guava single fruit image sample output by the image projection head.

[0070] The two projection vectors are represented as follows:

[0071] ;

[0072] in, These are the image samples of the i-th guava fruit. The two corresponding projection vectors, For image projection head, These are the parameters for the image projection head. Here, θ represents the feature encoder, and θ represents the parameters of the feature encoder.

[0073] Normalize the two projection vectors corresponding to each guava single fruit image sample to obtain two normalized results for each guava single fruit image sample:

[0074] ;

[0075] in, These are the image samples of the i-th guava fruit. The two corresponding normalization results are both identity norm eigenvectors. It is the Euclidean norm.

[0076] Using the two normalized results corresponding to each guava fruit image sample, the batch average contrast loss is calculated. This batch average contrast loss can be the InfoNCE loss, obtained by averaging the contrast losses of the guava fruit image samples in the batch. The contrast loss of guava fruit image samples in a batch of size N can be expressed as... And there are:

[0077] ;

[0078] in, It is the temperature coefficient, and sim is the similarity function.

[0079] Batch average comparative loss can be expressed as:

[0080] ;

[0081] in, The average loss is compared to the batch average.

[0082] By using batch average contrast loss, the feature encoder and image projector are jointly trained until a preset number of iterations are reached or the batch average contrast loss converges, resulting in an applicable feature encoder and image projector.

[0083] In this embodiment of the invention, an enhanced version of a guava single-fruit image is obtained by performing two random enhancements on the image samples. These enhanced versions are then input into a feature encoder and an image projector to obtain projection vectors. After normalization, the batch average contrastive loss is calculated, and the feature encoder and image projector are then jointly trained. This method enables the feature encoder to automatically learn a high-dimensional representation with high discriminative power that reflects the characteristics of different maturity stages of guava, overcoming the drawback of relying on a large amount of manual annotation, which is costly.

[0084] Based on the above embodiments, determining the spatial position of each guava fruit in a single image based on the guava fruit image patch and the depth information includes:

[0085] Based on the guava single fruit image block, determine the center pixel coordinates of the corresponding guava single fruit;

[0086] Based on the camera's intrinsic parameter matrix, the coordinates of the center pixel, and the depth information corresponding to the center pixel of the guava fruit, the position of the guava fruit in the camera coordinate system is calculated.

[0087] The spatial position of the guava fruit is calculated based on its position in the camera coordinate system.

[0088] Specifically, when determining the spatial location of each guava fruit in a single image, the center pixel coordinates of the corresponding guava fruit can be determined first using the guava fruit image patch:

[0089] ;

[0090] in, Let be the center pixel coordinates of the i-th guava fruit in a single image. For the i-th guava fruit image patch in a single image, for The total number of pixels within, for The coordinates of a single pixel in the image.

[0091] Subsequently, using the camera's intrinsic parameter matrix, the coordinates of the center pixel, and the depth information corresponding to the center pixel of a single guava fruit, the position of the guava fruit in the camera coordinate system is calculated:

[0092] ;

[0093] in, Let be the position vector of the i-th guava fruit in a single image in the camera coordinate system. This represents the depth information of the i-th guava fruit in a single image, in meters.

[0094] Calculate the spatial position of a single guava fruit using its position in the camera coordinate system:

[0095] ;

[0096] in, Let i be the spatial location of the i-th guava fruit in a single image. Let be the rigid body transformation matrix from the UAV coordinate system B to the world coordinate system W. Let be the rigid body transformation matrix from camera coordinate system C to UAV coordinate system B.

[0097] Generally speaking, from the coordinate system To coordinate system The rigid body transformation matrix can be expressed as:

[0098] ;

[0099] in, From coordinate system To coordinate system The rigid body transformation matrix, It is a 3×3 rotation matrix. coordinate system From the origin to the coordinate system The translation vector of the origin, It is a 1×3 zero row vector.

[0100] In this embodiment of the invention, the position of the guava fruit in the camera coordinate system is calculated by determining the center pixel coordinates of a single guava fruit image block, combining the camera's intrinsic parameters and the depth information corresponding to those coordinates, and then calculating the three-dimensional spatial position of the guava fruit in the world coordinate system by combining the camera's extrinsic parameters. This method can accurately map a two-dimensional image to three-dimensional physical space, providing high-precision target spatial positioning information for the subsequent planning and control of drones and robotic arms.

[0101] Based on the above embodiments, the step of sequentially controlling the drone to fly to the spatial location of each target guava fruit in the target fruit collection based on the picking access path includes: The current image captured by the camera during the control process is acquired and the current image is identified. When the target guava fruit and the robotic arm are present in the current image, the target center pixel coordinates of the target guava fruit and the end effector pixel coordinates of the robotic arm are determined. Calculate the image plane error based on the target center pixel coordinates and the end effector pixel coordinates; Based on the image plane error, the end effector speed of the robotic arm is calculated, and the end effector speed is mapped to the joint speed of the robotic arm; The robotic arm is controlled based on the joint speed.

[0102] Specifically, when controlling the drone, the system can acquire real-time images captured by the drone's onboard camera within the guava enclosure and perform image recognition. If the image identifies a target guava fruit and a robotic arm, the system determines the target center pixel coordinates of the guava fruit and the end effector pixel coordinates of the robotic arm based on the current image.

[0103] Subsequently, using the target center pixel coordinates of the guava fruit in the current image and the pixel coordinates of the end effector of the robotic arm, the image planar error is calculated:

[0104] ;

[0105] in, For image plane error, Let these be the components of the image plane error in the u and v directions. The target center pixel coordinates of a single guava fruit. These are the pixel coordinates of the end effector of the robotic arm.

[0106] The end effector velocity of the robotic arm is calculated using image plane errors. The end effector velocity can include linear velocity and angular velocity, and can be expressed as:

[0107] ;

[0108] in, For the end effector speed, Let be the image Jacobian matrix, and have , The rate of change of the image plane error. >0 represents the control gain. for The generalized inverse matrix.

[0109] Map the end effector speed to the joint speed of the robotic arm:

[0110] ;

[0111] in, For the joint speed of the robotic arm, Let Jacobian matrix be the value of the robotic arm. for The generalized inverse matrix.

[0112] Finally, the robotic arm is controlled by utilizing the joint speed of the robotic arm.

[0113] In this embodiment of the invention, the image plane error between the target center pixel coordinates of the guava fruit and the pixel coordinates of the end effector of the robotic arm is calculated, and the end effector speed is calculated and mapped to joint speed, thereby controlling the robotic arm. This method utilizes visual servoing technology to drive the pose adjustment of the end effector by continuously reducing the image plane error in a dynamic environment where the drone experiences slight shaking, achieving precise dynamic alignment of the robotic arm's end effector with the target guava fruit.

[0114] Based on the above embodiments, the harvest maturity benefit is determined based on the sum of the maturity weights of each candidate guava fruit in the candidate harvesting path, and the maturity weights of each candidate guava fruit are obtained based on the maturity level mapping of each candidate guava fruit.

[0115] The optimization problem satisfies a time constraint, which is that the total duration of the candidate picking paths is less than or equal to the total available time of the task. The total duration of the paths is equal to the sum of the flight times between two adjacent candidate guava fruits in the candidate picking paths and the sum of the picking times of each candidate guava fruit in the candidate picking paths.

[0116] Specifically, since the number of individual guava fruits in the greenhouse is much greater than the maximum number that can be harvested by a drone in one harvesting task, the purpose of solving the optimization problem is actually to select the most valuable set of individual guava fruits as the target harvesting fruit set within a limited time, and to generate the harvesting access path corresponding to the target harvesting fruit set according to the harvesting order of each target individual guava fruit in the target harvesting fruit set.

[0117] First, establish a candidate harvesting path. This path includes multiple candidate guava fruits, the number of which is the planned harvest quantity and can be set as needed. Each candidate guava fruit can be selected from the guava fruits contained in the greenhouse.

[0118] Candidate harvesting routes can be represented as:

[0119] ;

[0120] in, Here, M represents the candidate picking paths, and M represents the number of candidate guava fruits within each path. Let j be the index of the j-th candidate guava fruit in the candidate picking path. This is the set of indices for the individual guava fruits contained within the greenhouse.

[0121] The harvest maturity benefit can be determined by summing the maturity weights of individual guava fruits in each candidate harvesting path, and can be expressed as:

[0122] ;

[0123] in, The revenue based on the harvest maturity per unit of time. Let be the maturity weight of the j-th candidate guava fruit in the candidate picking path.

[0124] This can be obtained by mapping the maturity level of each candidate guava fruit, and can be represented as:

[0125] ;

[0126] in, Let j represent the maturity level of the j-th candidate guava fruit in the candidate harvesting path. For example, a mapping function. .

[0127] The optimization problem satisfies the time constraint that the total duration of candidate picking paths is less than or equal to the total available task time.

[0128] ;

[0129] in, Candidate picking routes, The total duration of the candidate picking routes. Total available time for the task. For drones to select candidate guava fruits from candidate harvesting paths Fly to candidate guava single fruit Flight duration, Candidate Guava Single Fruit The picking time is M, where M is the number of candidate guava fruits in the candidate picking path.

[0130] For any two guava fruits i and j, the flight time of the UAV from guava fruit i to guava fruit j can be expressed as:

[0131] ;

[0132] in, The flight time of the drone from Guava fruit i to Guava fruit j. Let i and j be the position vectors of guava fruits in the world coordinate system, respectively. This represents the average flight speed of the drone.

[0133] Based on this, the optimization problem can be expressed as:

[0134] .

[0135] In this embodiment of the invention, the time constraints that the optimization problem must satisfy are defined, and actual limitations such as battery life or operating time can be incorporated into the planning to ensure that the generated picking and access path can be executed smoothly within the working capacity of the UAV system, thereby improving the practicality of the system and the safety of the operation.

[0136] Based on the above embodiments, the step of controlling the robotic arm mounted on the drone to perform flexible harvesting of each target guava fruit includes:

[0137] Determine the desired pose and actual pose of the end effector of the robotic arm, and determine the desired contact force and actual contact force of the end effector of the robotic arm.

[0138] Based on the desired pose and the actual pose of the end effector, the pose error is calculated, and based on the desired contact force and the actual contact force of the end effector, the force error is calculated. Based on the posture error and the force error, the impedance equation is applied to control the robotic arm mounted on the UAV to flexibly harvest each target guava fruit.

[0139] Specifically, when controlling a drone-mounted robotic arm to flexibly harvest individual guava fruits, the desired pose, actual pose, desired contact force, and actual contact force of the robotic arm's end effector can be determined first. The desired pose refers to the three-dimensional spatial position and orientation of the robotic arm's end effector when it aligns with and grasps the target guava fruit under ideal conditions. The actual pose is the true three-dimensional position and six-dimensional pose vector of the robotic arm's end effector in physical space. The desired contact force refers to the safe grasping torque or force of the end effector set without damaging the fragile guava fruit. The actual contact force is the contact force or torque between the end effector and the target guava fruit, actually measured by sensors during the robotic arm's operation.

[0140] In actual operation, the actual contact force and pose of the robotic arm's end effector can be acquired in real time using multi-dimensional force sensors and pose sensors mounted on the end effector. Simultaneously, based on a pre-planned, finely tuned picking trajectory using visual servoing technology and set compliance parameters, the desired pose and contact force of the end effector can be obtained. By accurately acquiring these poses and contact forces, the relative spatial position and force of the robotic arm's end effector can be precisely monitored in real time, ensuring its alignment with the ideal safe operating state. This provides a precise data foundation for subsequent flexible compensation control.

[0141] Position error is used to characterize the degree of deviation between the actual position and orientation of the end effector of the robotic arm and the desired position and orientation, while force error is used to characterize the degree of deviation between the actual contact force of the end effector and the desired contact force of the end effector.

[0142] In practice, the difference between the actual pose of the end effector and its expected pose is calculated to obtain the pose error, which characterizes the deviation in three-dimensional position and attitude. Similarly, the difference between the actual contact force and the expected contact force of the end effector is calculated to obtain the force error, which characterizes the deviation in contact force. Accurately calculating the pose error and force error allows for precise quantification of the dynamic offset and abnormal force conditions of the robotic arm's end effector, effectively preventing grasping failure due to pose deviation or fruit damage due to excessive instantaneous force.

[0143] Subsequently, by utilizing the pose error and force error, and applying the impedance equation, the robotic arm mounted on the UAV is controlled to flexibly harvest each target guava fruit. The impedance equation is the core mathematical model of the impedance control mechanism, which equates the contact behavior between the robotic arm's end effector and the greenhouse environment to a physical dynamic system composed of virtual mass, virtual damping, and virtual springs, enabling the end effector to have the ability to resist impact and adapt flexibly.

[0144] The impedance equation can be expressed as:

[0145] ;

[0146] in, For the quality matrix, Here is the damping matrix. Here is the stiffness matrix. For force error, For pose error, The speed of pose error, This represents the acceleration due to the pose error.

[0147] Impedance equations can transform the mechanical rigid contact behavior of the end effector of a robotic arm into a flexible and compliant motion characteristic, enabling the robotic arm to adapt to minor external disturbances or different shapes and sizes of target fruits, thus greatly improving the compliance of the underlying control strategy.

[0148] Based on the impedance equation, by dynamically adjusting control parameters such as the mass matrix, damping matrix, and stiffness matrix in the impedance equation, the end effector of the robotic arm exhibits elastic, soft-hand behavior at the physical contact level. When the end effector approaches and contacts the target guava fruit, the impedance equation automatically and dynamically adjusts the three-dimensional pose of the end effector according to the sensed contact force, absorbing the slight swaying of the drone during aerial hovering operations and the impact of the fruit itself, thereby driving the end effector to safely pick the guava fruit with an adaptive, gentle gripping motion.

[0149] By using impedance equation-based control, it can effectively adapt to the fragile characteristics of soft fruits such as guava, and ensure that the gripping force at the end of the robotic arm is moderate even under complex greenhouse conditions, thus significantly reducing the rate of mechanical damage to the fruit during the harvesting process.

[0150] In this embodiment of the invention, by introducing an impedance control model to calculate the pose error and force error of the end effector in real time and constructing an impedance equation during the harvesting process of a drone-assisted robotic arm, the end effector of the robotic arm exhibits a soft and adaptive capability similar to a soft hand when it comes into contact with the target guava. This effectively overcomes the problem of guava fruit damage caused by slight shaking of the drone and rigid collisions with the environment, and realizes a safe, precise and autonomous flexible integrated harvesting operation for fragile crops.

[0151] Based on the above embodiments, obtaining the guava single fruit image block within a single image in the time-series image includes:

[0152] The single image is segmented to obtain multiple guava single-fruit masks;

[0153] Based on the multiple guava single-fruit masks, the single image is cropped to obtain the guava single-fruit image block.

[0154] By segmenting a single image into instances to obtain multiple guava single-fruit masks, and then cropping the single image accordingly, we can obtain guava single-fruit image blocks containing only a single fruit. This effectively eliminates interference from complex backgrounds such as leaves and branches, accurately extracts individual fruit regions, and provides high-quality input data for the subsequent feature encoder to learn purer fruit features.

[0155] like Figure 2 As shown, based on the above embodiments, this embodiment of the invention provides a guava ripeness visual detection and harvesting control device, comprising:

[0156] The acquisition module 21 is used to acquire time-series images from multiple perspectives captured by the camera mounted on the drone inside the greenhouse, and to acquire guava single fruit image blocks and depth information of the guava single fruit image blocks in a single image in the time-series images. Based on the guava single fruit image blocks and the depth information, the spatial position of each guava single fruit in the single image is determined.

[0157] The maturity determination module 22 is used to input the guava single fruit image block into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder, cluster the guava single fruit feature vector to obtain the category of each guava single fruit, and determine the maturity level corresponding to the category of each guava single fruit.

[0158] The optimization problem construction module 23 is used to construct an optimization problem based on the spatial location of each guava fruit and the maturity level, with the goal of maximizing the harvest maturity benefit per unit time, and solve the optimization problem to obtain the target set of harvestable fruits and the corresponding harvesting access path.

[0159] The picking control module 24 is used to control the drone to fly to the spatial position of each target guava fruit in the target fruit collection according to the picking access path, and to control the robotic arm on the drone to perform flexible picking of each target guava fruit.

[0160] Specifically, the functions of each module in the guava maturity visual detection and picking control device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0161] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the guava ripeness visual detection and harvesting control method provided in the above embodiments.

[0162] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the guava maturity visual detection and harvesting control method provided in the above embodiments.

[0164] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the guava ripeness visual detection and harvesting control method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visual detection and harvesting control of guava ripeness, characterized in that, include: The system acquires time-series images from multiple perspectives captured by a camera mounted on a drone inside a greenhouse, and obtains guava fruit image blocks and depth information of the guava fruit image blocks within a single image in the time-series images. Based on the guava fruit image blocks and the depth information, the spatial position of each guava fruit in the single image is determined. The guava single fruit image patch is input into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder. The guava single fruit feature vector is clustered to obtain the category of each guava single fruit, and the maturity level corresponding to the category of each guava single fruit is determined. Based on the spatial location of each guava fruit and the maturity level, an optimization problem is constructed with the goal of maximizing the harvest maturity benefit per unit time. The optimization problem is then solved to obtain the target set of harvestable fruits and the corresponding harvesting access path. Based on the harvesting access path, the drone is sequentially controlled to fly to the spatial location of each target guava fruit in the target harvesting fruit set, and the robotic arm on the drone is controlled to perform flexible harvesting of each target guava fruit.

2. The method for visual detection and harvesting control of guava maturity according to claim 1, characterized in that, The training steps of the feature encoder include: Each guava single fruit image sample in each batch was randomly enhanced twice to obtain two enhanced versions of each guava single fruit image sample; Two enhanced versions of each guava single fruit image sample are sequentially input into the feature encoder and the image projection head to obtain two projection vectors corresponding to each guava single fruit image sample output by the image projection head. Normalize the two projection vectors corresponding to each single guava fruit image sample to obtain two normalized results for each single guava fruit image sample, and calculate the batch average contrast loss based on the two normalized results for each single guava fruit image sample. The feature encoder and the image projector are jointly trained based on the batch average contrast loss.

3. The method for visual detection and harvesting control of guava maturity according to claim 1, characterized in that, Determining the spatial position of each guava fruit in a single image based on the guava fruit image patch and the depth information includes: Based on the guava single fruit image block, determine the center pixel coordinates of the corresponding guava single fruit; Based on the camera's intrinsic parameter matrix, the coordinates of the center pixel, and the depth information corresponding to the center pixel of the guava fruit, the position of the guava fruit in the camera coordinate system is calculated. The spatial position of the guava fruit is calculated based on its position in the camera coordinate system.

4. The method for visual detection and harvesting control of guava maturity according to claim 3, characterized in that, The step of controlling the drone to fly to the spatial location of each target guava fruit in the target fruit collection based on the picking access path includes: The current image captured by the camera during the control process is acquired and the current image is identified. When the target guava fruit and the robotic arm are present in the current image, the target center pixel coordinates of the target guava fruit and the end effector pixel coordinates of the robotic arm are determined. Calculate the image plane error based on the target center pixel coordinates and the end effector pixel coordinates; Based on the image plane error, the end effector speed of the robotic arm is calculated, and the end effector speed is mapped to the joint speed of the robotic arm; The robotic arm is controlled based on the joint speed.

5. The method for visual detection and harvesting control of guava maturity according to any one of claims 1-3, characterized in that, The optimization problem satisfies a time constraint, which is that the total path duration of the candidate picking paths is less than or equal to the total available task time. The total path duration is equal to the sum of the flight time between two adjacent candidate guava fruits in the candidate picking path and the sum of the picking time of each candidate guava fruit in the candidate picking path. The harvest maturity benefit is determined based on the sum of the maturity weights of each candidate guava fruit in the candidate harvesting path, and the maturity weights of each candidate guava fruit are obtained by mapping the maturity level of each candidate guava fruit.

6. The method for visual detection and harvesting control of guava maturity according to any one of claims 1-3, characterized in that, The process of controlling the robotic arm mounted on the drone to gently harvest each target guava fruit includes: Determine the desired pose and actual pose of the end effector of the robotic arm, and determine the desired contact force and actual contact force of the end effector of the robotic arm. Based on the desired pose and the actual pose of the end effector, the pose error is calculated, and based on the desired contact force and the actual contact force of the end effector, the force error is calculated. Based on the posture error and the force error, the impedance equation is applied to control the robotic arm mounted on the UAV to flexibly harvest each target guava fruit.

7. The method for visual detection and harvesting control of guava maturity according to any one of claims 1-3, characterized in that, The step of obtaining a single guava fruit image patch within a single image in the time-series image includes: The single image is segmented to obtain multiple guava single-fruit masks; Based on the multiple guava single-fruit masks, the single image is cropped to obtain the guava single-fruit image block.

8. A visual detection and harvesting control device for guava ripeness, characterized in that, include: The acquisition module is used to acquire time-series images from multiple perspectives captured by a camera mounted on a drone inside a greenhouse, and to acquire guava fruit image blocks and depth information of the guava fruit image blocks in a single image within the time-series images. Based on the guava fruit image blocks and the depth information, the spatial position of each guava fruit in the single image is determined. The maturity determination module is used to input the guava single fruit image block into the feature encoder to obtain the guava single fruit feature vector output by the feature encoder, cluster the guava single fruit feature vector to obtain the category of each guava single fruit, and determine the maturity level corresponding to the category of each guava single fruit. The optimization problem construction module is used to construct an optimization problem based on the spatial location of each guava fruit and the maturity level, with the goal of maximizing the harvest maturity benefit per unit time, and solve the optimization problem to obtain the target set of harvestable fruits and the corresponding harvesting access path. The harvesting control module is used to control the drone to fly to the spatial position of each target guava fruit in the target fruit collection according to the harvesting access path, and to control the robotic arm on the drone to perform flexible harvesting of each target guava fruit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the guava ripeness visual detection and harvesting control method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the guava ripeness visual detection and harvesting control method as described in any one of claims 1-7.

Images