Picking robot arm operation method and system based on visual feedback

Abstract

The application provides a picking mechanical arm operation method and system based on visual feedback, which first acquires a multi-dimensional visual perception set of a picking object, including shape contour, surface physical characteristic visual representation, connection part spatial distribution and growth environment dynamic interference; then constructs a mechanical arm cooperative decision model based on the multi-dimensional visual perception set, analyzes the stress distribution characteristics of the connection part, determines the optimal contact area parameters and the mechanical arm posture cooperative parameters; then generates a time sequence action instruction set containing action information of the contact, clamping and separation stages according to the above parameters; when the mechanical arm executes the instruction set, the dynamic visual feedback set of the picking scene is acquired in real time; finally, the input parameters of the cooperative decision model are updated based on the feedback, the instruction set parameters are adjusted, the picking operation of the mechanical arm is completed under the premise of maintaining the integrity of the picking object, and the intelligent level of the picking mechanical arm and the picking operation quality are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method and system for operating a harvesting robotic arm based on visual feedback. Background Technology

[0002] In the field of agricultural harvesting automation, the application of robotic arms is a key means to achieve efficient and precise harvesting. However, existing robotic arm operation methods have many limitations.

[0003] On the one hand, traditional methods have a relatively limited perspective on the harvested objects, typically focusing only on their basic shape or location information, making it difficult to comprehensively and meticulously understand their characteristics. For example, they lack effective perception of factors such as differences in the hardness of the object's skin, the morphology of its connection structure with the parent plant, and dynamic interference from the growth environment. This results in an inability to accurately handle the harvested object based on its actual condition, easily leading to damage or harvesting failure.

[0004] On the other hand, existing operating methods lack flexibility and adaptability in decision-making and motion control. The robotic arm's motion commands are often pre-set fixed patterns, unable to dynamically adjust based on the real-time contact state between the harvested object and the end effector, the object's displacement trend, and changes in environmental disturbances. This makes it difficult for the robotic arm to guarantee the stability and accuracy of harvesting operations in complex and ever-changing harvesting scenarios, failing to meet the high demands of modern agriculture for harvesting quality and efficiency. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for operating a harvesting robotic arm based on visual feedback, the method comprising:

[0006] A multi-dimensional visual perception set of the harvested object is obtained. The multi-dimensional visual perception set includes the morphological contour information of the harvested object, the visual representation information of the surface physical properties, the spatial distribution information of the connection parts, and the dynamic interference information of the growth environment. The visual representation information of the surface physical properties reflects the difference in the hardness of the harvested object's epidermis through the change of image grayscale gradient. The spatial distribution information of the connection parts is used to describe the connection structure morphology between the harvested object and the parent body.

[0007] Based on the multi-dimensional visual perception set, a collaborative decision-making model for the robotic arm is constructed. The stress distribution characteristics of the connection parts of the picking object are analyzed by the collaborative decision-making model. Combined with the structural parameters of the end effector of the picking robotic arm, the optimal contact area parameters of the picking object and the posture coordination parameters of the robotic arm are determined. The optimal contact area parameters need to match the stress-weak area of ​​the connection part, and the posture coordination parameters of the robotic arm need to keep the end effector and the optimal contact area at a preset angle deviation.

[0008] The robotic arm time-series motion instruction set is generated based on the optimal contact area parameters and the robotic arm posture coordination parameters. The time-series motion instruction set includes buffer motion trajectory information of the contact stage, force gradient change information of the clamping stage, and angle compensation motion information of the separation stage. The motion instructions of each stage are associated with the deformation trend of the picking object by visual prediction.

[0009] The picking robotic arm is controlled to execute the time-sequence action instruction set. During the execution, a dynamic visual feedback set of the picking scene is acquired in real time. The dynamic visual feedback set includes information on the change of contact state between the picking object and the end effector, information on the displacement trend of the picking object relative to the parent body, and information on the motion trajectory of environmental interference sources.

[0010] The input parameters of the robotic arm collaborative decision-making model are updated based on the dynamic visual feedback set, and the stage connection parameters and action intensity parameters of the time-series action instruction set are adjusted so that the picking robotic arm can complete the picking operation while maintaining the integrity of the picking object.

[0011] In another aspect, embodiments of the present invention also provide an operating system for a harvesting robotic arm based on visual feedback, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0012] Based on the above, this invention, through acquiring a multi-dimensional visual perception set of the harvested object, including its shape and contour, visual representation of surface physical properties, spatial distribution of connection parts, and dynamic interference from the growth environment, comprehensively and accurately depicts the characteristics of the harvested object. The robotic arm collaborative decision-making model constructed based on this multi-dimensional visual perception set can deeply analyze the stress distribution characteristics of the connection parts of the harvested object. Combined with the structural parameters of the robotic arm's end effector, it scientifically and rationally determines the optimal contact area parameters and the robotic arm's posture coordination parameters. Based on the time-series action instruction set generated according to these parameters, it plans the action information for the contact, gripping, and separation stages in detail, and correlates these with the visually predicted deformation trend of the harvested object, making the robotic arm's movements smoother and more coordinated. During execution, a dynamic visual feedback set of the harvesting scene is acquired in real time, and the input parameters of the collaborative decision-making model are updated promptly based on this dynamic visual feedback set. The stage connection parameters and action intensity parameters of the time-series action instruction set are adjusted, enabling the harvesting robotic arm to dynamically adjust according to the actual situation. This allows for efficient completion of the harvesting operation while maintaining the integrity of the harvested object, significantly improving the intelligence level of the harvesting robotic arm and the quality and efficiency of the harvesting operation. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the execution flow of the visual feedback-based harvesting robotic arm operation method provided in an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of exemplary hardware and software components of the visual feedback-based harvesting robotic arm operating system provided in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a visual feedback-based harvesting robotic arm operation method according to an embodiment of the present invention. The following is a detailed description of the visual feedback-based harvesting robotic arm operation method.

[0016] Step S110: Obtain a multi-dimensional visual perception set of the harvested object. The multi-dimensional visual perception set includes the shape and outline information of the harvested object, the visual representation information of the surface physical properties, the spatial distribution information of the connection parts, and the dynamic interference information of the growth environment. The visual representation information of the surface physical properties reflects the difference in the hardness of the harvested object's skin through the change of image grayscale gradient. The spatial distribution information of the connection parts is used to describe the connection structure morphology between the harvested object and the parent body.

[0017] This embodiment uses cherry picking in a greenhouse as an application scenario. During cherry picking operations, multi-view color image acquisition devices and depth image acquisition devices deployed at the end of the robotic arm, on the greenhouse roof support, and on the side columns are used to collaboratively acquire a multi-dimensional visual perception set of the target cherry. Among them, morphological contour information is used to fully present the overall shape of the cherry, the smoothness of the contour edges, and whether there are deformities; surface physical property visual representation information indirectly reflects the hardness differences of different parts of the cherry skin through the variation of gray-level gradients in different areas of the image, such as the difference in hardness between smooth areas and areas with fruit spots; spatial distribution information of connection parts is specifically for the connection structure between the cherry stem and the branch, including the growth angle of the stem, changes in thickness, and the shape of the connection point with the branch; dynamic interference information of the growth environment covers dynamic factors that may interfere with the robotic arm picking operation, such as airflow generated by the greenhouse ventilation system, worker movement, and the movement of other operating machinery.

[0018] Step S111: Dynamically switch and acquire images of the picking scene. Adjust the viewing angle parameters of the image acquisition device according to the proportion of the occluded area of ​​the picking object in the initial acquired image. When the proportion of the occluded area exceeds the preset ratio, trigger the viewing angle lifting or side shifting operation to obtain an unobstructed initial visual image of the picking object.

[0019] The multi-view image acquisition device is activated to acquire initial view images of the harvesting scene containing the target cherries. Since cherries often grow in clusters and are frequently obstructed by leaves, overlapping branches, and other cherry fruits, occlusion analysis is required for the initial acquired images. By calculating the proportion of the occluded area to the target cherry area in the initial acquired image, it is determined whether the initial viewpoint meets the requirements for subsequent feature extraction. When this proportion exceeds a preset proportion, it indicates that key features of the target cherry may be obstructed from the initial viewpoint, and the viewpoint adjustment mechanism of the image acquisition device must be triggered immediately. Depending on the specific location of the occluded area on the target cherry, either a viewpoint elevation or lateral shift operation is selected, and images are repeatedly adjusted and acquired until an initial visual image of the unobstructed cherry with an occluded area proportion lower than the preset proportion is obtained.

[0020] Step S1111: Acquire initial view images of the picking scene to obtain initial images of the picking object. Use an image segmentation algorithm to separate the picking object area from the background area and mark the occlusion area in the background area that obscures the picking object.

[0021] The system controls a color image acquisition device mounted on the shoulder of a robotic arm to capture images of the harvesting scene from a default perspective, obtaining an initial image containing the target cherries. Using an image segmentation algorithm based on the HSV color space, the cherry region and background region are separated from the initial image based on the cherry's red hue range, saturation, and brightness characteristics. Within the separated background region, the Canny edge detection algorithm is used to extract edge features. Combined with the green hue of the leaves, the gray-brown hue of the branches, and morphological features, areas such as leaves and branches obscuring the target cherries are identified. A pixel-level boundary marking and internal filling marking algorithm is then used to mark these obscured areas for subsequent calculation of the obscuration percentage.

[0022] Step S1112: Calculate the proportion of the obstructed area in the picking target area to obtain the proportion of the obstructed area.

[0023] The image pixel statistics module counts the total number of pixels in the occluded area marked in step S1111, as well as the total number of pixels in the separated target cherry area. A division operation is then performed, dividing the total number of pixels in the occluded area by the total number of pixels in the target cherry area to obtain the occlusion ratio. This ratio is expressed as a decimal, reflecting the proportion of the occluded area within the target cherry area.

[0024] Step S1113: When the proportion of the occluded area is less than or equal to a preset ratio, the initial view image is used as the initial visual image of the picking object; when the proportion of the occluded area is greater than the preset ratio, the view adjustment mechanism of the image acquisition device is activated.

[0025] The occlusion area ratio calculated in step S1112 is compared with a preset ratio. If the ratio is less than or equal to the preset ratio, it indicates that the key features of the target cherry in the initial view image are not severely occluded, and the initial view image can be directly determined as the initial visual image of the cherry. If the ratio is greater than the preset ratio, a view adjustment start command is sent to the drive control module of the image acquisition device to activate the view adjustment mechanism composed of the pitch adjustment mechanism and the horizontal translation mechanism.

[0026] Step S1114: If the occluded area is located below the object being picked, control the image acquisition device to perform a viewing angle raising operation. After each raising by a preset angle, acquire an image and calculate the proportion of the occluded area in the newly acquired image until the proportion of the occluded area is less than or equal to the preset ratio.

[0027] The image region localization algorithm determines the location distribution of the occluded areas marked in step S1111 within the target cherry region. If the occluded areas are mainly concentrated in the area below the target cherries, the pitch adjustment mechanism of the image acquisition device is driven to perform a viewing angle increase operation. After each increase of a preset angle, the image acquisition device is controlled to immediately acquire a new scene image, and the image segmentation, occluded area marking, and proportion calculation process of steps S1111 to S1112 are repeated on the new image. If the proportion of the occluded area in the new image is still greater than the preset proportion, the pitch adjustment mechanism is controlled to continue performing the increase operation until the proportion of the occluded area in the acquired image is less than or equal to the preset proportion.

[0028] Step S1115: If the occluded area is located to the side of the object being picked, control the image acquisition device to perform a side-shift operation. After each side-shift by a preset distance, acquire an image and calculate the proportion of the occluded area in the new image until the proportion of the occluded area is less than or equal to the preset ratio.

[0029] If the image region localization algorithm determines that the occlusion area is mainly distributed on the side of the target cherry, the horizontal translation mechanism of the image acquisition device is driven to perform a side-shift operation in the direction away from the occlusion. After each side-shift by a preset distance, the image acquisition device immediately acquires a new scene image and re-performs the image segmentation, occlusion area marking, and proportion calculation steps on the new image. If the proportion of the occlusion area in the new image does not meet the preset requirement, the horizontal translation mechanism continues to perform the side-shift operation until the acquired image meets the occlusion area proportion requirement.

[0030] Step S1116: Determine the image that finally meets the requirement of the proportion of the occluded area as the initial visual image of the picking object, and record the corresponding viewing angle parameters of the image acquisition device.

[0031] When the percentage of the occluded area in the acquired image is less than or equal to a preset ratio, the image is officially designated as the initial visual image of the cherry. Simultaneously, the parameter reading module of the image acquisition device obtains and records the viewing angle parameters of the device, including the tilt angle of the pitch adjustment mechanism, the lateral shift distance of the horizontal translation mechanism, the lens focal length, and the aperture size. These parameters are stored in a local database so that they can be directly retrieved when re-acquiring images of the target cherry without requiring further viewing angle adjustments.

[0032] Step S112: Perform multi-scale contour extraction on the initial visual image of the picking object to obtain the morphological contour information of the picking object.

[0033] The initial visual image of the cherry determined in step S1116 is preprocessed. First, a multi-scale Gaussian filtering algorithm is used to smooth the image at different filtering scales, removing Gaussian noise and salt-and-pepper noise. At each filtering scale, the Canny edge detection algorithm is applied to extract the edge features of the cherry: at a small filtering scale, the focus is on capturing the subtle edge features formed by the skin protrusions and fruit point distribution on the cherry outline; at a large filtering scale, the overall outline shape of the cherry is obtained, including the approximate outline of the fruit top and stem. The edge features extracted at different scales are integrated using a feature fusion algorithm, and a non-maximum suppression algorithm is used to remove redundant edges and false edges, finally obtaining a cherry outline pixel coordinate sequence composed of continuous pixel coordinates. This cherry outline pixel coordinate sequence is the morphological outline information of the cherry.

[0034] Step S113: Perform grayscale gradient partitioning calculation on the initial visual image of the harvested object, divide the initial visual image of the harvested object into multiple pixel blocks, calculate the gradient change rate of grayscale values ​​in each pixel block, combine it with the pre-calibrated grayscale-hardness mapping relationship, divide the skin hardness level region according to the magnitude of the gradient change rate, and form visual characterization information of surface physical properties. The larger the gradient change rate, the higher the skin hardness.

[0035] The initial visual image of the cherry was converted to an 8-bit grayscale image using a grayscale conversion algorithm. The grayscale image was then uniformly divided into multiple non-overlapping pixel blocks according to a preset rectangular size. The size of each pixel block was set based on the image resolution and cherry size to ensure that each pixel block could reflect the characteristics of a local area of ​​the cherry skin. For each pixel block, the grayscale gradient values ​​of each pixel within the block were calculated in the horizontal and vertical directions using the Sobel operator. The average gradient values ​​of all pixels within the block were then calculated, and this average value was defined as the grayscale gradient change rate of the pixel block. A pre-calibrated grayscale-hardness mapping relationship was used. This mapping relationship was established through image acquisition and hardness testing of cherry skins of different ripeness levels and varieties, recording the skin hardness levels corresponding to different grayscale gradient change rates. Based on the grayscale gradient change rate of each pixel block, the corresponding skin hardness level is matched from the mapping relationship. A pseudo-color coding algorithm is used to fill and mark pixel blocks of different hardness levels with different colors to form a visual representation of surface physical properties. The pixel block with the larger grayscale gradient change rate corresponds to the higher skin hardness level and the darker the marking color.

[0036] Step S114: Perform feature enhancement processing on the connection parts in the initial visual image of the picking object, use a frequency domain filtering algorithm to enhance the grayscale difference between the connection parts and the main body of the picking object, extract the contour curve and spatial coordinates of the connection parts, and combine the position information of the parent body of the picking object to generate spatial distribution information of the connection parts.

[0037] For the initial visual image of cherries, the connection area between the cherry body and the stem is first transformed to the frequency domain using a Fourier transform algorithm, resulting in a frequency domain spectrogram. In the frequency domain, a high-pass filter is used to remove background noise corresponding to low-frequency components and uniform grayscale areas of the cherry body, while preserving and enhancing the edge features of the connection area corresponding to high-frequency components, thereby increasing the grayscale difference between the connection area and the cherry body. The filtered frequency domain spectrogram is then transformed back to the spatial domain using an inverse Fourier transform, resulting in an image with enhanced connection features. In this enhanced image, an active contour model algorithm is used to extract the complete contour curve of the stem. Using stereo vision matching technology combined with depth information obtained from a depth image acquisition device, the three-dimensional spatial coordinates of each feature point on the stem contour curve are calculated. Simultaneously, an image segmentation algorithm is used to separate the cherry parent branch from the background region, determining the three-dimensional spatial position information of the branch. The three-dimensional spatial coordinates of the stem, the contour curve parameters, and the branch position information are integrated to generate spatial distribution information of the connection area, including stem length, diameter variation, coordinates of the connection point with the branch, and the angle between the stem and the branch.

[0038] Step S115: Acquire continuous frame images of the picking scene, analyze the dynamic regions in adjacent frame images other than the picking object, calculate the motion direction and speed parameters of the dynamic regions, mark the dynamic interference sources that may affect the operation of the robotic arm, and form dynamic interference information of the growth environment.

[0039] The image acquisition device continuously captures images of the harvesting scene at a preset frame rate, which must meet the requirements for capturing the motion characteristics of dynamic interference sources. Inter-frame difference operations are performed on two consecutively acquired frames to obtain a difference image. A threshold segmentation algorithm is used to extract regions whose grayscale value changes exceed a preset threshold from the difference image; these are identified as dynamic regions. After excluding potential target cherry areas within these dynamic regions, morphological processing is performed on the remaining dynamic regions to remove small noise areas, resulting in complete dynamic interference candidate regions. By tracking the positional changes of these candidate regions across multiple frames, optical flow is used to calculate the motion direction vector and velocity magnitude of the region. Combining the robotic arm's motion range parameters with the preset operating trajectory, it is determined whether the candidate region is likely to enter the robotic arm's operating space. If entry is possible, the region is marked as a dynamic interference source, and its motion direction, velocity magnitude, and current position coordinates are recorded, forming dynamic interference information for the growing environment.

[0040] Step S116: Integrate the morphological contour information, surface physical property visual representation information, spatial distribution information of connection parts, and dynamic interference information of the growth environment to form a multi-dimensional visual perception set of the harvested object.

[0041] A standardized data storage structure is constructed, comprising four data fields corresponding to morphological contour information, visual representation information of surface physical properties, spatial distribution information of connection sites, and dynamic interference information of the growth environment. The morphological contour pixel coordinate sequence obtained in step S112, the pseudo-color coded hardness level image and corresponding pixel block hardness parameters generated in step S113, the three-dimensional coordinates and structural parameters of the connection sites obtained in step S114, and the motion parameters of the dynamic interference source recorded in step S115 are respectively filled into the corresponding data fields. A timestamp synchronization algorithm is used to calibrate the acquisition timestamps of the above four types of information to ensure consistency of different information in the time dimension and avoid information misalignment caused by acquisition delays. The integrated data undergoes integrity verification, eliminating invalid data and erroneous parameters, ultimately forming a complete multi-dimensional visual perception set of cherries.

[0042] Step S120: Construct a robotic arm collaborative decision-making model based on the multi-dimensional visual perception set. Analyze the stress distribution characteristics of the connection parts of the picking object through the collaborative decision-making model. Combine the structural parameters of the end effector of the picking robotic arm to determine the optimal contact area parameters of the picking object and the robotic arm posture collaborative parameters. The optimal contact area parameters need to match the stress-weak area of ​​the connection part, and the robotic arm posture collaborative parameters need to ensure that the end effector maintains a preset angle deviation from the optimal contact area.

[0043] Using the multi-dimensional visual perception set of cherries integrated in step S116 as the basic input data, a collaborative decision-making model for the robotic arm is constructed. This model employs an architecture that integrates a deep learning sub-model and a finite element analysis sub-model. The deep learning sub-model uses a combination of convolutional neural networks and recurrent neural networks to extract the geometric and physical characteristics of the connection points from the multi-dimensional visual perception set. The finite element analysis sub-model is used to analyze the stress distribution characteristics of the fruit stalk connection points based on the extracted features. The structural parameters of the end effector of the harvesting robotic arm are input into the collaborative decision-making model. Through feature matching and optimization algorithms within the model, the optimal contact area parameters that match the weak stress areas of the fruit stalk are determined. Simultaneously, the robotic arm posture coordination parameters that maintain a preset angular deviation between the end effector and the optimal contact area are calculated. These posture coordination parameters encompass the angles of each joint of the robotic arm and the spatial posture of the end effector.

[0044] Step S121: Extract the spatial distribution information of the connection parts and the visual representation information of the surface physical properties from the multi-dimensional visual perception set, convert the morphological contour information of the connection parts into a three-dimensional spatial coordinate sequence, and combine the epidermal hardness level in the visual representation information of the surface physical properties to construct a three-dimensional physical property mapping model of the connection parts.

[0045] Spatial distribution information of connecting parts and visual representation information of surface physical properties are extracted from the data fields of a multi-dimensional visual perception set. The pixel coordinates of the fruit stalk contour curve in the spatial distribution information of connecting parts are transformed using camera calibration parameters, converting the pixel coordinates into a three-dimensional spatial coordinate sequence in the world coordinate system. This three-dimensional spatial coordinate sequence contains the three-dimensional position coordinates of all feature points of the fruit stalk from its connection with the cherry body to its connection with the branch. The epidermal hardness level of each pixel block in the fruit stalk region is extracted from the visual representation information of surface physical properties. Based on the pixel block position corresponding to each feature point in the three-dimensional spatial coordinate sequence, a corresponding epidermal hardness level parameter is assigned to each feature point. Using 3D modeling software, a 3D mesh model of the fruit stalk is constructed based on the above 3D spatial coordinate sequence and hardness level parameters. Different physical attribute labels are assigned to regions with different hardness levels in the model, forming a 3D physical property mapping model that reflects the geometric shape and physical property distribution of different locations of the fruit stalk.

[0046] Step S122: Based on the three-dimensional physical property mapping model, the stress distribution of the connection part under different contact forces is simulated using the finite element analysis method, and the coordinates of the contact area corresponding to the minimum stress value are extracted as candidate contact area parameters.

[0047] The three-dimensional physical property mapping model of the fruit stalk constructed in step S121 is imported into the finite element analysis software. The model is then meshed using the software's mesh generation function. The mesh density is adjusted according to the geometric complexity of the fruit stalk, with finer meshes used at points where the fruit stalk diameter changes and at the connection points with branches. Based on the hardness grade parameters in the three-dimensional physical property mapping model, corresponding material mechanics parameters are assigned to each finite element mesh element. Multiple sets of different contact force parameters are set, each set including the magnitude and direction of the contact force. The magnitude of the contact force covers the conventional clamping force range of the end effector, and the direction of action covers the axial, radial, and multiple tilting directions of the fruit stalk. For each set of contact force parameters, contact loads and constraints are applied in the finite element analysis software, and the stress calculation module is run to obtain the stress distribution results of the fruit stalk under the action of the contact force. The region with the minimum stress value is identified from the stress distribution results, and the three-dimensional spatial coordinates of this region are extracted. These coordinates are used as candidate contact area parameters.

[0048] Step S1221: Import the three-dimensional coordinate sequence of the connection part in the three-dimensional physical property mapping model into the finite element analysis software, and divide the finite element mesh of the connection part.

[0049] Launch the finite element analysis software. Using the software's standard data interface, import the 3D coordinate sequence of the fruit stalk from the 3D physical property mapping model into the software's modeling environment in ASCII format to generate the fruit stalk's geometric model. Based on the fruit stalk's geometric dimensions and analysis accuracy requirements, select the tetrahedral mesh type to automatically mesh the geometric model. Set the maximum and minimum size thresholds for the mesh cells to ensure that the mesh cell size meets the stress calculation accuracy requirements without causing computational inefficiency due to an excessive number of mesh cells. After meshing, use the software's mesh quality check tool to inspect the mesh for quality indicators such as twist, aspect ratio, and volume ratio. Re-mesh or locally refine areas that do not meet the quality standards until all mesh cells meet the preset quality requirements.

[0050] Step S1222: Based on the surface hardness level in the visual characterization information of surface physical properties, query the pre-established hardness level-material parameter mapping table, and assign the corresponding elastic modulus and Poisson's ratio parameters to each element of the finite element mesh. The higher the hardness level, the greater the elastic modulus and the smaller the Poisson's ratio.

[0051] A hardness grade-material parameter mapping table, pre-stored in the database, is retrieved. This table was established by conducting tensile and compression tests on cherry stem samples with different hardness grades, recording the elastic modulus and Poisson's ratio parameters of the stem material corresponding to each hardness grade. The epidermal hardness grade of each pixel block in the stem region is extracted from the visual characterization information of surface physical properties. Based on the position of the finite element mesh element on the stem geometry model, the hardness grade of the pixel block corresponding to each mesh element is determined. The corresponding elastic modulus and Poisson's ratio parameters are then retrieved from the mapping table based on this hardness grade. Using the material property assignment function of the finite element analysis software, the retrieved parameters are assigned to the corresponding mesh elements, achieving a spatially differentiated distribution of material parameters in the finite element model. Mesh elements with higher hardness grades are assigned a larger elastic modulus parameter and a smaller Poisson's ratio parameter.

[0052] Step S1223: Set multiple sets of different contact force parameters. Each set of contact force parameters includes the magnitude of the contact force and the direction of the contact force. The direction of the contact force covers multiple possible contact angles of the connection part.

[0053] Based on the technical parameter manual of the end effector of the harvesting robot, the minimum and maximum clamping force ranges of the end effector are obtained. Multiple contact force magnitudes are then evenly selected within this range to form a contact force magnitude sequence. Considering the geometric characteristics of the fruit stalk, the possible directions of the contact force are determined, including the tensile force direction along the stalk axis, the radial pressure direction perpendicular to the stalk axis, and the tilting directions at multiple different angles to the axial direction, ensuring that the direction of action covers all possible angles of contact between the end effector and the fruit stalk. Each contact force magnitude value is combined with each direction of action to form multiple sets of independent contact force parameters, each set of parameters uniquely corresponding to a combination of contact force magnitude and direction of action.

[0054] Step S1224: Perform finite element stress calculation on each set of contact force parameters to obtain the stress distribution cloud map of the connection part under the action of the contact force, and extract the three-dimensional coordinates of the region with the minimum stress value in the stress distribution cloud map.

[0055] For each set of contact force parameters, a contact model was constructed in the finite element analysis software. The direction of action in the contact force parameters was converted into a force vector direction recognizable by the software, and the magnitude of the contact force was converted into a corresponding load value. This contact load was then applied to the preset contact area of ​​the finite element model of the fruit stalk. Simultaneously, based on the actual connection state between the fruit stalk and the branch, fixed constraints were set at the connection point, restricting all degrees of freedom at that location. The stress calculation solver of the finite element analysis software was then activated. Based on the material parameters of the mesh elements, the applied load, and the constraints, the solver performed iterative finite element calculations to obtain the stress distribution of the fruit stalk under the contact force, and visualized it as a stress distribution cloud map. Using the software's region selection tool, the continuous region with the minimum stress value was located and selected in the stress distribution cloud map. The coordinate extraction function was used to obtain the three-dimensional spatial coordinates of multiple feature points within this region, with each feature point's coordinates containing components in the X, Y, and Z directions.

[0056] Step S1225: Perform cluster analysis on the coordinates of the stress minimum region corresponding to multiple sets of contact force parameters, group coordinates that are close to each other into the same candidate region, and calculate the center coordinates of each candidate region.

[0057] The three-dimensional coordinates of feature points corresponding to the minimum stress values ​​of all contact force parameters are compiled to construct a coordinate dataset. A clustering analysis algorithm is then used to process this dataset. First, the Euclidean distance between any two coordinate points in the dataset is calculated, using distance as a similarity metric. Clustering thresholds and distance thresholds are set, and coordinate points with Euclidean distances less than the thresholds are grouped into the same cluster, with each cluster corresponding to a candidate contact region. For all coordinate points within each cluster, the average values ​​of their X, Y, and Z coordinate components are calculated. These average values ​​are combined to form a new three-dimensional coordinate system, which represents the center coordinates of the candidate contact region.

[0058] Step S1226: Based on the center coordinates of the candidate regions and the structural characteristics of the connection parts, select the center coordinates of the stress-weak areas of the connection parts as candidate contact region parameters.

[0059] The three-dimensional physical property mapping model of the fruit stalk constructed in step S121 is retrieved. Combined with the fruit stalk structural parameters in the spatial distribution information of the connection points, the specific location of the center coordinates of each candidate contact area on the fruit stalk is analyzed to determine whether this location belongs to a stress-weak area of ​​the fruit stalk. Stress-weak areas of the fruit stalk typically include areas near the connection between the cherry body and the fruit stalk, areas where the diameter of the fruit stalk suddenly decreases, etc. By comparing the center coordinates of the candidate areas with the spatial range of the stress-weak areas, the center coordinates of the candidate areas located within the stress-weak areas are selected. The selected center coordinates are compiled into a coordinate list as candidate contact area parameters.

[0060] Step S123: Obtain the structural parameters of the end effector of the harvesting robot arm, input the structural parameters into the three-dimensional physical property mapping model, and calculate the matching degree between the candidate contact area parameters and the end effector. The structural parameters include the clamping surface shape parameters, contact area parameters, and elastic coefficient parameters of the end effector.

[0061] The structural parameters of the end effector are read through the control system interface of the harvesting robot. These parameters include the gripping surface shape parameters (radius of curvature, chamfer dimensions, etc.) describing the geometry of the gripping surface; the contact area parameter (maximum effective contact area when the end effector contacts the harvested object); and the elastic modulus parameter (elastic modulus of the gripping surface material, reflecting its deformation capacity). These structural parameters are converted to a data format compatible with the three-dimensional physical property mapping model and input into the model, constructing a simplified geometric model of the end effector's gripping surface. For each candidate contact area parameter, the center coordinates are used to simulate the contact process between the end effector's gripping surface and the candidate area using a collision detection algorithm. Indicators such as the percentage of contact area between the gripping surface and the candidate area, and the uniformity of pressure distribution at the contact point are calculated. A weighted summation algorithm is then used to comprehensively calculate these indicators, yielding the matching degree between each candidate contact area and the end effector.

[0062] Step S124: When the matching degree meets the preset requirements, the candidate contact area parameters are determined as the optimal contact area parameters; when the matching degree does not meet the preset requirements, the range of the candidate contact area parameters is adjusted, and the stress distribution and matching degree are recalculated until the optimal contact area parameters that meet the requirements are obtained.

[0063] The matching degree value of each candidate contact area is compared with a preset matching degree threshold. If the matching degree value of a candidate contact area is greater than or equal to the matching degree threshold, it indicates that the candidate area has good structural compatibility with the end effector. The center coordinates and corresponding area range of the candidate contact area are determined as the optimal contact area parameters. If the matching degree values ​​of all candidate contact areas are less than the matching degree threshold, the distance threshold of the cluster analysis in step S1225 is increased, or the number of contact force parameter groups is increased. The stress calculation, cluster analysis, and matching degree calculation process in steps S1224 to S123 are re-executed to generate new candidate contact area parameters and calculate their matching degree. The above process is repeated until candidate contact area parameters with matching degrees that meet the preset requirements are obtained, and these are determined as the optimal contact area parameters.

[0064] Step S125: Extract dynamic interference information of the growth environment from the multi-dimensional visual perception set, analyze the potential impact of interference sources on the movement trajectory of the robotic arm, and determine the interference avoidance posture reference parameters of the robotic arm.

[0065] From the dynamic interference information field of the growth environment in the multi-dimensional visual perception set, parameters such as the current position coordinates, motion direction vector, and motion speed of the dynamic interference source are extracted. The kinematic model of the harvesting robot arm is invoked. This kinematic model includes parameters such as the length and range of motion of each joint of the robot arm, and can calculate the spatial trajectory of the end effector based on the joint angles. Based on the motion parameters of the interference source, a motion trajectory prediction algorithm is used to predict the continuous position coordinates of the interference source over a future period, forming the predicted motion trajectory of the interference source. The preset motion trajectory of the robot arm and the predicted motion trajectory of the interference source are spatially superimposed and analyzed to determine whether the two trajectories intersect or have a distance less than a safe threshold. Based on the analysis results, the spatial direction range in which the robot arm needs to avoid the interference source is determined. Based on this range, interference avoidance angle constraints are set for each joint of the robot arm, i.e., interference avoidance posture reference parameters. These interference avoidance posture reference parameters clearly define the upper and lower limits of the angle that each joint must not exceed during movement.

[0066] Step S126: Combining the optimal contact area parameters and the interference avoidance posture reference parameters, calculate the angle parameters of each joint of the robotic arm and the spatial posture parameters of the end effector, so that the end effector can avoid the interference source while maintaining a preset angle deviation from the optimal contact area, thus forming the robotic arm posture coordination parameters.

[0067] The three-dimensional coordinates in the optimal contact area parameters are used as the target position coordinates of the end effector and input into the robot arm inverse kinematics solution module. This module uses the target position coordinates as constraints and introduces joint angle constraints from the interference avoidance attitude reference parameters. It solves for the target angle parameters required for each joint of the robot arm to move from its current angle to the target position using inverse kinematic equations. The target angle of each joint must be within the angle range specified by the interference avoidance attitude reference parameters. Based on the preset angle deviation requirements between the end effector and the optimal contact area (i.e., the angle range between the end effector's gripping surface and the surface of the optimal contact area), the module calculates the spatial attitude parameters of the end effector, such as pitch, yaw, and roll angles, ensuring that this angle meets the preset deviation requirements. The solved target angle parameters of each joint are integrated with the spatial attitude parameters of the end effector to form a parameter set, resulting in the robot arm attitude coordination parameters.

[0068] Step S130: Generate a time-series motion instruction set for the robotic arm based on the optimal contact area parameters and the robotic arm posture coordination parameters. The time-series motion instruction set includes buffer motion trajectory information for the contact phase, force gradient change information for the clamping phase, and angle compensation motion information for the separation phase. The motion instructions for each phase are associated with the deformation trend of the picking object as predicted by vision.

[0069] Based on the target position in the optimal contact area parameters and the joint angles and end effector posture in the robotic arm posture coordination parameters, motion parameters are generated for each of the three temporal stages of the robotic arm operation: contact, gripping, and separation. By analyzing the morphological contour information and visual representation information of the surface physical characteristics of the cherry, the potential deformation trends of the cherry in each stage of contact, gripping, and separation are predicted, such as slight indentations of the skin during contact and slight deformation of the fruit during gripping. Based on these deformation trends, trigger conditions and parameter relationships are set between the motion commands of each stage to ensure that the deformation result of the previous stage does not affect the execution accuracy of the subsequent stage. The motion parameters and relationships of the three stages are integrated to form a time-series motion command set for the robotic arm that includes time series, motion parameters, and force parameters.

[0070] Step S131: Based on the optimal contact area parameters, determine the contact start coordinate and contact end coordinate of the end effector, calculate the spatial distance between the contact start coordinate and the contact end coordinate, and combine the joint motion speed limit in the robot arm posture coordination parameters to plan the buffer motion trajectory of the contact stage. The speed of the buffer motion trajectory gradually decreases from the initial speed to the contact speed, forming the buffer motion trajectory information of the contact stage.

[0071] The three-dimensional spatial coordinates of the current end effector of the robotic arm are determined as the contact start coordinates, and the center coordinates in the optimal contact area parameters are determined as the contact end coordinates. Based on the spatial coordinate calculation method, the coordinate differences between the contact start and contact end coordinates in the X, Y, and Z directions are calculated, and then the straight-line spatial distance between them is calculated using square root and square root operations. Motion speed limit parameters for each joint are extracted from the robotic arm's posture coordination parameters; these parameters define the maximum angular velocity of each joint during movement. Using the joint motion speed limits as constraints, and combined with the calculated spatial distance, the motion trajectory of the end effector from the contact start coordinate to the contact end coordinate is planned. This trajectory uses a smooth curve transition to avoid abrupt changes in motion. During trajectory planning, a speed variation law is set so that the end effector's speed starts from an initial speed and gradually decreases as the movement distance increases, reaching the contact end coordinate precisely at the contact speed, forming a buffered motion characteristic. The coordinate sequence, speed variation sequence, and motion time parameters of this trajectory are organized to form the buffered motion trajectory information for the contact phase.

[0072] Step S1311: Determine the coordinates of the starting point and ending point of the motion during the contact phase. The coordinates of the starting point are the coordinates of the end effector position before the robotic arm performs the contact action, and the coordinates of the ending point are the contact start coordinates in the optimal contact area parameters.

[0073] The position feedback interface of the robotic arm controller reads the three-dimensional spatial coordinates of the end effector in real time before the robotic arm performs the contact action. These three-dimensional spatial coordinates are the starting coordinates of the motion in the contact phase. A preset contact start coordinate is extracted from the optimal contact area parameters. This contact start coordinate is the initial position coordinate of the end effector when it begins to contact the cherry stem, and it is determined as the ending coordinate of the motion in the contact phase. Both the starting and ending coordinates are described using the world coordinate system to ensure consistency of the coordinate reference.

[0074] Step S1312: Calculate the straight-line distance between the coordinates of the starting point and the ending point of the motion. Combined with the maximum motion speed of the robotic arm, determine the initial speed of the contact phase. The initial speed shall not exceed a preset proportion of the maximum motion speed of the robotic arm.

[0075] Using the formula for calculating the distance between two points in space, the X, Y, and Z components of the coordinates of the starting point and ending point of the motion are subtracted respectively. Each difference is squared and summed, and then the square root of the sum is taken to obtain the straight-line distance between the two points. The maximum speed of the end effector is obtained from the performance parameter document of the robotic arm; this maximum speed is the maximum movement speed of the robotic arm in an unloaded state. Based on a preset proportional coefficient, the maximum speed of the robotic arm is multiplied by the proportional coefficient to obtain the initial speed upper limit. The initial speed during the contact phase is set to a value not exceeding this upper limit.

[0076] Step S1313: Based on the surface hardness grade in the visual characterization information of the physical properties of the object being picked, and combined with the pre-calibrated hardness-maximum permissible contact speed relationship, determine the contact speed. The lower the surface hardness grade, the lower the contact speed.

[0077] From the visual representation information of the physical properties of the object's surface, the pixel block corresponding to the optimal contact area is located, and the skin hardness level of that pixel block is read. A pre-stored hardness-maximum permissible contact speed database is invoked. This database, established through extensive experiments, records the maximum permissible contact speed of the end effector corresponding to different skin hardness levels. Based on the extracted skin hardness level, a matching query is performed in the database to obtain the corresponding maximum permissible contact speed, which is then determined as the contact speed for the contact stage. Notably, there is a negative correlation between skin hardness level and maximum permissible contact speed; that is, the lower the skin hardness level, the lower the retrieved contact speed.

[0078] Step S1314: Use an exponential decay function to plan the speed change curve. The speed starts from the initial speed and decays exponentially with the increase of the movement distance. When the end effector reaches the end point coordinate of the movement, the speed decreases to the contact speed.

[0079] An exponential decay function is chosen as the mathematical model for velocity planning. This function is expressed as a velocity decreasing exponentially with increasing travel distance. The initial velocity determined in step S1312 is used as the initial input value of the function, and the contact velocity determined in step S1313 is used as the target output value. By adjusting the decay coefficient of the exponential decay function, the function curve satisfies the following conditions: when the end effector starts moving from the starting point coordinate and the travel distance is zero, the velocity is the initial velocity; when the travel distance reaches the straight-line distance between the starting and ending points, the velocity decays to the contact velocity. The decay coefficient is optimized using a function fitting algorithm to ensure a smooth velocity change curve without abrupt changes.

[0080] Step S1315: Based on the speed change curve and the movement distance, calculate the movement time of the contact phase, divide the movement time into multiple time intervals, and calculate the position coordinates and speed parameters of the actuator at the end of each time interval.

[0081] Based on the velocity variation curve and travel distance, the total motion time required for the end effector to complete the contact phase is calculated through integral calculation. The essence of integral calculation is to accumulate the relationship between velocity and distance. The total motion time is evenly divided into multiple equal time intervals, the duration of which is set according to the motion control accuracy requirements. For each time interval, the average velocity of the end effector within that interval is calculated based on the velocity variation curve. The average velocity is multiplied by the time interval duration to obtain the displacement of the end effector within that time interval. Based on the coordinates of the starting point of motion, the displacement of each time interval, and the direction of motion, the position coordinates of the end effector in the X, Y, and Z directions at the end of each time interval are calculated sequentially, and the average velocity parameter within that time interval is recorded.

[0082] Step S1316: Connect the position coordinates of each time interval in chronological order to form the buffer motion trajectory of the contact phase, and record the position coordinates and corresponding velocity parameters of each point on the buffer motion trajectory as buffer motion trajectory information.

[0083] The end effector position coordinates at the end of each time interval calculated in step S1315 are connected sequentially in chronological order to form a continuous three-dimensional spatial curve, which is the buffer motion trajectory during the contact phase. To ensure the smoothness of the trajectory, an interpolation algorithm is used to supplement the intermediate transition point coordinates between adjacent position coordinates. All position coordinates on the trajectory (including the coordinates of the time interval endpoints and the interpolation point coordinates) are arranged in chronological order, and the velocity parameters at each position coordinate are recorded accordingly, forming a dataset containing the position-velocity-time correspondence. This dataset is the buffer motion trajectory information.

[0084] Step S132: Extract the skin hardness level corresponding to the optimal contact area from the surface physical characteristics visual representation information of the multi-dimensional visual perception set, determine the clamping force range of the end effector according to the skin hardness level, plan the force change curve of the clamping stage, the force gradually increases from the initial force at contact to the preset clamping force, and the rate of increase is negatively correlated with the skin hardness level, forming the force gradient change information of the clamping stage.

[0085] In the surface physical property visual representation information of the multi-dimensional visual perception set, the pixel block corresponding to the optimal contact area is located according to the coordinate range of the optimal contact area, and the skin hardness level of the pixel block is extracted. A pre-established correspondence table of skin hardness level and clamping force range is consulted. This table was established through clamping experiments on cherry stems of different hardness levels, recording the minimum safe clamping force and maximum allowable clamping force of the end effector for each hardness level, which together constitute the clamping force range. The small force applied by the end effector at the end of the contact phase is set as the initial clamping force, and a suitable value within the clamping force range is selected as the preset clamping force. Starting from the initial force and ending at the preset clamping force, a force change curve for the clamping phase is planned. The shape of the curve is determined according to the skin hardness level: the higher the skin hardness level, the faster the force increases; the lower the skin hardness level, the slower the force increases, i.e., the increase rate is negatively correlated with the skin hardness level. The time parameters, force parameters, and change rate parameters of the force change curve are organized to form the force gradient change information for the clamping phase.

[0086] Step S133: Based on the spatial distribution information of the connection parts, analyze the separation direction of the connection parts, combine the angle parameters in the robot arm posture coordination parameters, determine the initial angle and target angle of the separation stage, calculate the angle change and the required time, plan the angle compensation motion trajectory of the separation stage, so that the end effector adjusts the angle along the separation direction of the connection parts during the separation process, and forms the angle compensation motion information of the separation stage.

[0087] Analyzing the structural parameters of the fruit stalk and the coordinates of the connection point between the fruit stalk and the branch in the spatial distribution information of the connection parts, and combining this with the natural characteristics of cherry growth, the optimal separation direction for the end effector to apply force when the cherry separates from the branch is determined. This optimal separation direction is typically the tangent direction at the connection point between the fruit stalk and the branch. The pitch, yaw, and roll angles of the end effector at the end of the gripping phase are extracted from the robotic arm's posture coordination parameters, and these angles are determined as the initial angles for the separation phase. Based on the separation direction requirements, the target angle that the end effector needs to adjust to is calculated, ensuring that the force direction of the end effector is consistent with the separation direction. The difference between the initial angle and the target angle is calculated to obtain the angle change. Based on the angle adjustment speed parameters of the robotic arm's end effector, the separation time required to complete this angle change is calculated. Based on the angle change, separation time, and separation direction, the angle compensation motion trajectory of the end effector is planned, ensuring a smooth and continuous angle adjustment process. The angle parameters, time parameters, and motion direction parameters of this trajectory are organized to form the angle compensation motion information for the separation phase.

[0088] Step S134: Analyze the connection conditions between each stage of the time-sequenced action command. When the end effector of the contact stage reaches the contact termination coordinate and the clamping force reaches the preset initial force, the clamping stage action command is triggered. When the clamping force reaches the preset clamping force and the connection part shows a tendency to separate, the separation stage action command is triggered, thus forming a stage connection rule.

[0089] Logical analysis was performed on the motion commands for the contact, clamping, and separation stages to clarify the triggering conditions for the transitions between each stage. For the transition between the contact and clamping stages, a dual triggering condition was established: first, the position sensor detected that the end effector had reached the contact termination coordinate; second, the force sensor detected that the force applied by the end effector had reached the preset initial force. When both conditions were met simultaneously, the motion command for the clamping stage was automatically triggered. Similarly, for the transition between the clamping and separation stages, a dual triggering condition was established: first, the force sensor detected that the clamping force had reached the preset clamping force; second, image analysis technology detected slight displacement, deformation, or other separation trend characteristics at the connection point between the fruit stalk and the branch. When both conditions were met simultaneously, the motion command for the separation stage was triggered. The above transition conditions, triggering logic, and command switching rules were compiled into a standardized stage transition rule document, which clarifies the triggering sequence and logical relationships of the motion commands for each stage.

[0090] Step S135: Integrate the buffer motion trajectory information of the contact phase, the force gradient change information of the clamping phase, the angle compensation motion information of the separation phase, and the phase connection rules to generate a time-sequenced action instruction set for the robotic arm.

[0091] Create a time-series instruction set data framework containing time axis fields, motion parameter fields, force parameter fields, attitude parameter fields, and control logic fields. Fill the motion parameter field with the position coordinate sequence and velocity change sequence from the buffer motion trajectory information of the contact phase, and associate them with the corresponding time axis nodes. Fill the force parameter field with the force change curve and force increase rate parameter from the force gradient change information of the clamping phase, and bind them to the corresponding time intervals. Fill the attitude parameter field with the angle change parameter and trajectory coordinate sequence from the angle compensation motion information of the separation phase, and match them with the corresponding time nodes. Embed the trigger conditions and instruction switching logic from the phase transition rules into the control logic field, establishing trigger associations between parameters of each phase. Perform logical validation on the integrated data framework, checking the continuity of the time axis, the consistency of parameters, and the rationality of trigger conditions, correcting any conflicts or omissions, and finally generating a complete time-series motion instruction set for the robotic arm.

[0092] Step S140: Control the picking robotic arm to execute the time-series action instruction set, and acquire the dynamic visual feedback set of the picking scene in real time during the execution process. The dynamic visual feedback set includes information on the change of contact state between the picking object and the end effector, information on the displacement trend of the picking object relative to the parent body, and information on the motion trajectory of environmental interference sources.

[0093] The generated time-series motion instruction set for the robotic arm is sent to the main controller of the harvesting robotic arm via industrial Ethernet. The main controller parses the time axis and parameter information in the instruction set and sends motion control signals to the servo drivers of each joint of the robotic arm, driving the joints to move along a preset trajectory and speed, sequentially performing contact, gripping, and separation actions. Simultaneously with the robotic arm executing the instructions, image acquisition devices deployed at the end effector of the robotic arm and on the greenhouse support are activated to continuously acquire images of the harvesting scene at a high frame rate. The acquired images are processed in real time to extract information such as changes in the contact state between the cherries and the end effector, the displacement trend of the cherries relative to the branches, and the motion trajectory of dynamic interference sources. This information is then synchronously integrated according to timestamps to form a dynamic visual feedback set.

[0094] Step S141: Control the picking robotic arm to execute each stage of action according to the stage connection rules of the time-sequence action instruction set, and simultaneously start the image acquisition device to perform high-frequency image acquisition, with the acquisition frequency being consistent with the stage change frequency of the robotic arm's action.

[0095] The robotic arm's main controller, based on the phase transition rules in the time-sequential motion instruction set, sequentially calls the motion parameters for the contact phase, gripping phase, and separation phase, controlling the servo drivers to drive the movement of each joint. When the contact phase begins, a start command is synchronously sent to the image acquisition device, setting the acquisition frequency to be the same as the phase change frequency of the robotic arm's motion. That is, within the time interval of the robotic arm completing one phase transition, the image acquisition device completes exactly one complete image acquisition cycle. For example, if the transition cycle from the contact phase to the gripping phase is a certain duration, the acquisition frequency of the image acquisition device is set to the reciprocal of that duration, ensuring that a corresponding scene image is acquired after each phase transition, achieving synchronization between motion execution and image acquisition.

[0096] Step S142: Perform contact state recognition on the high-frequency acquired image, use image difference algorithm to calculate pixel changes in the contact area between the end effector and the picking object, determine the area change and position offset of the contact area, and form contact state change information.

[0097] Preprocessing is performed on high-frequency acquired consecutive frame images. First, the color images are converted to grayscale images using a grayscale conversion algorithm, and then median filtering is used to remove salt-and-pepper noise from the images. Image difference operations are performed on two adjacent preprocessed frames to obtain difference images. A threshold segmentation algorithm is used to extract regions from the difference images whose grayscale values ​​change beyond a threshold; these regions are the contact areas between the end effector and the cherry. The number of pixels in the contact areas across multiple consecutive frames is counted, and the rate of change in pixel count is calculated to determine the area change of the contact areas. Using a template matching algorithm, the contact areas in the first frame are used as templates to locate the positions of the contact areas in subsequent frames. The X and Y axis offsets in the image coordinate system are calculated, and then converted to three-dimensional position offsets in the world coordinate system using camera calibration parameters. The area change rate, three-dimensional position offsets, and corresponding timestamps of the contact areas are compiled to form contact state change information.

[0098] Step S143: Track and identify the connection parts in the high-frequency acquired images, extract the spatial coordinate changes of the connection parts in adjacent frame images, calculate the displacement distance and displacement direction of the picking object relative to the mother body, and form the displacement trend information of the picking object relative to the mother body.

[0099] In each frame of the frequently acquired images, a deep learning-based object detection model is used to identify the connection points between cherry stems and branches. This model, trained on a large number of cherry connection images, accurately outputs the bounding box coordinates of the connection points. The center pixel coordinates of the connection points are extracted based on the bounding box coordinates. Combined with depth information obtained from the depth image acquisition device, the pixel coordinates are converted into three-dimensional spatial coordinates in the world coordinate system using a camera perspective transformation algorithm. The difference between the three-dimensional spatial coordinates of the connection points in two adjacent frames is calculated to obtain the coordinate change. The displacement distance of the cherry fruit relative to the branch is calculated based on the magnitude of the coordinate change, and the displacement direction is determined based on the direction vector of the coordinate change. By linearly fitting the displacement distance and displacement direction of multiple consecutive frames, the displacement trend over a future period is predicted. The displacement distance, displacement direction, predicted trend, and timestamp are integrated to form the displacement trend information of the cherry fruit relative to the parent plant.

[0100] Step S144: Track the dynamic interference sources in the high-frequency acquired images, use the Kalman filter algorithm to predict the subsequent motion trajectory of the interference sources, mark the interference sources that may intersect with the motion trajectory of the robotic arm, and form the motion trajectory information of the environmental interference sources.

[0101] In high-frequency acquired images, based on the color, shape, and motion characteristics of dynamic interference sources, a background subtraction algorithm is used to separate the dynamic interference source region. The center pixel coordinates of this region are extracted and converted into three-dimensional position coordinates in the world coordinate system. These three-dimensional position coordinates are then input into a Kalman filter algorithm to construct the motion state equation and observation equation of the interference source. The current motion state of the interference source, including position, velocity, and acceleration, is estimated through a prediction-update iterative process. Based on the estimated motion state, the three-dimensional position coordinates of the interference source at multiple future time points are predicted, and these coordinates are connected to form the predicted motion trajectory of the interference source. The current motion trajectory parameters of the robotic arm are retrieved, and the spatial distance between them and the predicted motion trajectory of the interference source is calculated. If the distance between the two is less than a safety threshold at any time point, the interference source is marked as a high-risk interference source. The historical position coordinates, predicted motion trajectory, risk level markers, and timestamps of the interference source are compiled to form the motion trajectory information of the environmental interference source.

[0102] Step S145: Integrate the contact state change information, the displacement trend information of the picking object relative to the mother body, and the motion trajectory information of the environmental interference source to form a dynamic visual feedback set of the picking scene.

[0103] A data structure for the dynamic visual feedback set is constructed, comprising a contact state subset, a displacement trend subset, and an interference trajectory subset. The contact state change information obtained in step S142 is entered into the contact state subset in timestamp order, including parameters such as area change rate and position offset. The displacement trend information of the picking object relative to the parent body generated in step S143 is entered into the displacement trend subset, covering displacement distance, direction, and predicted trend data. The motion trajectory information of environmental interference sources obtained in step S144 is entered into the interference trajectory subset, including historical coordinates, predicted trajectory, and risk level information. A timestamp synchronization algorithm is used to time-calibrate the data in the three subsets, ensuring that different information at the same time point corresponds to each other. The integrated data undergoes a validity check, removing outliers and invalid data, ultimately forming the dynamic visual feedback set for the picking scene.

[0104] Step S150: Update the input parameters of the robotic arm collaborative decision-making model based on the dynamic visual feedback set, and adjust the stage connection parameters and action intensity parameters of the time-series action instruction set so that the picking robotic arm can complete the picking operation while maintaining the integrity of the picking object.

[0105] The contact state change information, displacement trend information, and interference trajectory information from the dynamic visual feedback set are converted into an input format compatible with the robotic arm collaborative decision-making model, updating the contact state parameters, displacement prediction parameters, and interference risk parameters in the model. Based on the updated parameters, the model recalculates the optimal contact area correction value, clamping force adjustment value, and posture compensation value, outputting an instruction adjustment scheme. According to this scheme, parameters such as stage transition time, force increase rate, and angle change amplitude in the time-series motion instruction set are adjusted to ensure that the robotic arm's movements adapt to the dynamic changes in the picking scenario, completing the picking operation while avoiding problems such as cherry skin damage and stem breakage.

[0106] Step S151: Extract contact state change information from the dynamic visual feedback set. When the position offset of the contact area exceeds a preset threshold, convert the position offset into posture correction parameters in the robotic arm collaborative decision-making model and update the input parameters of the robotic arm collaborative decision-making model.

[0107] The 3D position offset of the contact area is extracted from the contact state subset of the dynamic visual feedback set, and compared with a preset position offset threshold. If the offset is greater than the threshold, it indicates that the contact position between the end effector and the cherry deviates from the optimal contact area, and attitude correction is required. Using an inverse kinematics transformation algorithm, the 3D position offset is converted into angle correction values ​​for each joint of the robotic arm; these angle correction values ​​are the attitude correction parameters. The attitude correction parameters are input into the robotic arm collaborative decision-making model, replacing the original attitude parameters in the model, thus updating the model's input parameters and enabling the model to make decisions based on the current contact state.

[0108] Step S152: Extract the displacement trend information of the picking object relative to the parent body from the dynamic visual feedback set, calculate the matching degree between the displacement trend and the separation stage action in the time-series action instruction set, and when the matching degree is lower than the preset value, input the displacement trend parameter into the collaborative decision model and recalculate the angle compensation parameter of the separation stage.

[0109] Displacement trend parameters, such as displacement direction and velocity, of the picking object are extracted from the displacement trend subset of the dynamic visual feedback set. Action parameters for the separation stage are retrieved from the time-series action instruction set, including preset separation direction and preset angle change rate. A vector similarity calculation method is used to calculate the similarity between the displacement direction in the displacement trend parameters and the preset separation direction, and simultaneously calculate the matching degree between the displacement velocity and the motion velocity corresponding to the preset angle change rate. The overall matching degree is obtained by combining both. If the matching degree is lower than a preset value, the displacement trend parameters are input into the robotic arm collaborative decision-making model. The model, combining the morphological characteristics of the cherry and the structure of the connecting parts, recalculates the angle compensation parameters for the separation stage, such as the target angle and angle change amount.

[0110] Step S153: Extract the motion trajectory information of the environmental interference source from the dynamic visual feedback set. When the distance between the interference source and the motion trajectory of the robotic arm is less than the safety threshold, input the speed and direction parameters of the interference source into the collaborative decision-making model and adjust the interference avoidance angle in the collaborative parameters of the robotic arm posture.

[0111] The real-time position, velocity, and direction of motion of high-risk interference sources are extracted from the interference trajectory subset of the dynamic visual feedback set. The shortest spatial distance between the real-time position of the interference source and each point on the current trajectory of the robotic arm is calculated. If this distance is less than a safety threshold, it indicates a collision risk for the robotic arm. The velocity and direction of motion of the interference source are input into the robotic arm's collaborative decision-making model. Based on the kinematic constraints and obstacle avoidance priorities of the robotic arm, the model recalculates the upper and lower limits of the interference avoidance angles for each joint, adjusting the interference avoidance angles in the robotic arm's posture coordination parameters to ensure the robotic arm avoids the interference source during movement.

[0112] Step S154: Based on the updated collaborative decision-making model input parameters, calculate the connection time difference of each stage of the time-sequence action instruction set. When the connection time between the contact stage and the clamping stage exceeds the preset range, shorten or extend the buffer movement time of the contact stage and adjust the stage connection parameters.

[0113] Using the updated input parameters of the robotic arm collaborative decision-making model, the model recalculates the actual motion time of the contact phase and the start-up preparation time of the gripping phase; the difference between the two is the phase transition time difference. This transition time difference is compared with a preset transition time range. If it exceeds the range, it indicates a delay or overlap in the transition between the two phases. By adjusting the speed variation curve of the buffer motion trajectory in the contact phase, if the transition time is too long, the average speed of the buffer motion is increased, shortening the total motion time of the contact phase; if the transition time is too short, the average speed of the buffer motion is decreased, extending the total motion time of the contact phase. Through these adjustments, the phase transition time difference is brought within the preset range, and the phase transition parameters in the time-sequenced motion instruction set are updated.

[0114] Step S155: Based on the corrected clamping force parameters output by the collaborative decision-making model, adjust the force increase rate in the force gradient change information during the clamping stage. When the object being picked shows a deformation trend, reduce the force increase rate. Based on the corrected angle compensation parameters, adjust the angle change rate during the separation stage to ensure that the separation action is synchronized with the displacement trend of the object being picked.

[0115] The corrected gripping force parameters, including the corrected maximum gripping force and the force stability range, are extracted from the output of the robotic arm's collaborative decision-making model. These corrected parameters are compared with those in the original force gradient change information during the gripping phase to calculate the force difference. Simultaneously, image analysis technology is used to detect whether the cherry surface exhibits deformation trends such as dents or wrinkles. If deformation trends are present, the force increase rate in the force gradient change information is reduced, regardless of the force difference, to prevent further deformation. If no deformation trends are present, the rate is adjusted according to the force difference: the rate is increased when the force difference is positive and decreased when it is negative. Furthermore, the ratio of the angle change in the corrected angle compensation parameters to a preset time is calculated to obtain a new angle change rate. This rate of angle change during the separation phase is adjusted to ensure that the rhythm of the separation action is consistent with the displacement trend of the cherry being picked.

[0116] For example, step S1551: extract the corrected clamping force parameter from the output of the collaborative decision-making model, wherein the corrected clamping force parameter includes the maximum clamping force and the stable range of clamping force.

[0117] Through the output interface of the robotic arm collaborative decision-making model, the corrected clamping force parameter calculated by the model is read. The clamping force parameter consists of two parts: one is the maximum clamping force, which is the upper limit of the force that the end effector must not exceed during the clamping process; the other is the stable range of clamping force, which is the range of force that the end effector should maintain after the clamping action stabilizes. The minimum value of this force range is greater than the minimum clamping threshold of the cherry skin, and the maximum value is less than the damage force threshold of the cherry.

[0118] Step S1552: Compare the corrected maximum clamping force with the maximum clamping force in the original clamping stage force gradient change information, and calculate the force difference.

[0119] The original maximum clamping force parameter is extracted from the force gradient change information during the clamping phase of the time-series action instruction set, and then compared numerically with the corrected maximum clamping force. A subtraction operation is used: the corrected maximum clamping force is subtracted from the original maximum clamping force to obtain the force difference. If the difference is positive, the upper limit of the clamping force needs to be increased; if the difference is negative, the upper limit of the clamping force needs to be decreased; if the difference is zero, no adjustment to the upper limit of the force is required.

[0120] Step S1553: When the corrected maximum clamping force is greater than the original maximum clamping force, extract the deformation of the picking object from the dynamic visual feedback set. If the deformation is lower than the first preset threshold, increase the force increase rate so that the clamping force reaches the corrected maximum clamping force more quickly. If the deformation is higher than the first preset threshold but lower than the second preset threshold, maintain the original force increase rate and extend the time to reach the maximum clamping force.

[0121] When the force difference is positive, the cherry's deformation parameter is extracted from the displacement trend subset of the dynamic visual feedback set. This deformation parameter is calculated by the change in the cherry's outline in the image. The deformation parameter is compared with a first preset threshold and a second preset threshold: if the deformation parameter is lower than the first preset threshold, it indicates that the cherry's current deformation is small, and the force increase rate in the force gradient change information can be increased to shorten the time for the clamping force to reach the corrected maximum clamping force; if the deformation parameter is higher than the first preset threshold but lower than the second preset threshold, to avoid further deformation, the original force increase rate is kept unchanged, and the clamping time is extended to allow the force to reach the corrected maximum value.

[0122] Step S1554: When the corrected maximum clamping force is less than the original maximum clamping force, immediately reduce the force increase rate. If the current clamping force has exceeded the corrected maximum clamping force, trigger the force callback operation to gradually reduce the clamping force from the current value to the corrected maximum clamping force. The callback rate is determined according to the material deformation resistance parameter corresponding to the skin hardness level of the harvested object. The lower the hardness level, the slower the callback rate.

[0123] When the force difference is negative, the force increase rate during the clamping phase is immediately adjusted, reducing it to a certain proportion of the original rate to prevent the clamping force from continuing to rise rapidly. Simultaneously, a force sensor detects the current actual clamping force; if the current force exceeds the corrected maximum clamping force, a force callback operation is triggered. The cherry skin hardness grade is extracted from the visual characterization information of the surface physical properties, and a pre-established hardness grade-material deformation resistance mapping table is consulted to obtain the corresponding material deformation resistance parameter. The force callback rate is set according to this parameter; the lower the hardness grade, the weaker the material's deformation resistance, and the slower the callback rate is set, to avoid the cherry slipping due to a sudden drop in force or skin damage due to excessively rapid callback.

[0124] Step S1555: Based on the adjusted force increase rate or pullback rate, replan the force change curve during the clamping phase so that the force change curve is within the range of the corrected maximum clamping force, and the deformation of the object being picked does not exceed the allowable range during the change.

[0125] Using the revised maximum clamping force as the upper limit and the current clamping force as the starting point, a linear interpolation method is used to re-plan the force change curve during the clamping phase, based on the adjusted force increase rate or pullback rate. During the planning process, deformation monitoring data of the cherry from the dynamic visual feedback set is incorporated to ensure that the force value corresponding to each time point of the force change curve is within the range of the revised maximum clamping force, and that the corresponding deformation does not exceed the preset allowable deformation range. If a node in the planned curve may cause deformation to exceed the tolerance, the rate at that node is further adjusted until the curve meets the requirements.

[0126] Step S1556: Replace the original force gradient change information of the clamping stage with the adjusted force change curve, and update the clamping stage action parameters of the time-sequence action instruction set.

[0127] The redesigned force variation curve is exported as a standardized parameter file, containing force and velocity values ​​for each time interval. In the time-series motion instruction set, the force gradient change information field for the clamping phase is located, the original force variation curve data is deleted, and the new parameter file is imported to replace the force gradient change information. Simultaneously, the phase connection trigger condition parameters related to clamping force in the instruction set are updated to ensure that the connection logic matches the adjusted force variation curve, thus updating the clamping phase motion parameters in the time-series motion instruction set.

[0128] Step S156: Integrate the adjusted stage connection parameters and motion intensity parameters into the time-series motion instruction set, generate an updated time-series motion instruction set, and control the picking robotic arm to continue execution until the picking operation is completed.

[0129] Collect the stage connection parameters adjusted in step S154, such as the stage connection time and connection triggering conditions, as well as the motion intensity parameters adjusted in step S155, such as the force increase rate and angle change rate. Replace the corresponding original parameters with these parameters according to the data structure of the time-series motion instruction set. Simultaneously, use a logic verification algorithm to check the compatibility of the new parameters with other unadjusted parameters to ensure the overall logical coherence of the instruction set. After successful verification, generate an updated time-series motion instruction set and send it to the robotic arm's main controller. The main controller immediately switches to execute the updated instruction set, controlling the robotic arm to continue completing the remaining picking actions until the cherries are completely separated from the branches and remain intact, at which point the picking operation ends.

[0130] Figure 2 The illustration shows exemplary hardware and software components of a visual feedback-based harvesting robotic arm operating system 100, which can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used on the visual feedback-based harvesting robotic arm operating system 100 and to perform the functions in this application.

[0131] The visual feedback-based harvesting robotic arm operating system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the visual feedback-based harvesting robotic arm operation method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0132] For example, the visual feedback-based harvesting robotic arm operating system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the visual feedback-based harvesting robotic arm operating system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The visual feedback-based harvesting robotic arm operating system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0133] For ease of explanation, only one processor is described in the visual feedback-based harvesting robotic arm operating system 100. However, it should be noted that the visual feedback-based harvesting robotic arm operating system 100 of this application may also include multiple processors, and therefore the steps executed by one processor as described in this application may also be executed jointly by multiple processors or individually. For example, if the processor of the visual feedback-based harvesting robotic arm operating system 100 executes steps A and B, it should be understood that steps A and B may also be executed jointly by two different processors or individually by one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0134] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned visual feedback-based picking robotic arm operation method is implemented.

[0135] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for operating a harvesting robotic arm based on visual feedback, characterized in that, The method includes: Obtain a multi-dimensional visual perception set of the harvested object; Based on the multi-dimensional visual perception set, a collaborative decision-making model for the robotic arm is constructed. The stress distribution characteristics of the connection parts of the picking object are analyzed by the collaborative decision-making model. Combined with the structural parameters of the end effector of the picking robotic arm, the optimal contact area parameters of the picking object and the posture coordination parameters of the robotic arm are determined. The optimal contact area parameters need to match the stress-weak area of ​​the connection part, and the posture coordination parameters of the robotic arm need to keep the end effector and the optimal contact area at a preset angle deviation. The robotic arm time-series motion instruction set is generated based on the optimal contact area parameters and the robotic arm posture coordination parameters. The time-series motion instruction set includes buffer motion trajectory information of the contact stage, force gradient change information of the clamping stage, and angle compensation motion information of the separation stage. The motion instructions of each stage are associated with the deformation trend of the picking object by visual prediction. The picking robotic arm is controlled to execute the time-sequence action instruction set. During the execution, a dynamic visual feedback set of the picking scene is acquired in real time. The dynamic visual feedback set includes information on the change of contact state between the picking object and the end effector, information on the displacement trend of the picking object relative to the parent body, and information on the motion trajectory of environmental interference sources. The input parameters of the robotic arm collaborative decision-making model are updated based on the dynamic visual feedback set, and the stage connection parameters and action intensity parameters of the time-series action instruction set are adjusted so that the picking robotic arm can complete the picking operation while maintaining the integrity of the picking object. The acquisition of the multi-dimensional visual perception set of the harvested object includes: The harvesting scene is dynamically captured by switching perspectives. The perspective parameters of the image acquisition device are adjusted according to the proportion of the occluded area of ​​the harvesting object in the initial captured image. When the proportion of the occluded area exceeds the preset ratio, the perspective is raised or shifted to the side to obtain an initial visual image of the harvesting object without occlusion. Multi-scale contour extraction is performed on the initial visual image of the harvested object to obtain the morphological contour information of the harvested object; The initial visual image of the harvested object is divided into multiple pixel blocks by performing gray-level gradient partitioning calculation. The gradient change rate of gray values ​​in each pixel block is calculated and combined with the pre-calibrated gray-level-hardness mapping relationship. The surface hardness level region is divided according to the magnitude of the gradient change rate to form visual characterization information of surface physical properties. The larger the gradient change rate, the higher the surface hardness. Feature enhancement processing is performed on the connection parts in the initial visual image of the picking object. A frequency domain filtering algorithm is used to enhance the grayscale difference between the connection parts and the main body of the picking object. The contour curve and spatial coordinates of the connection parts are extracted. Combined with the position information of the parent body of the picking object, the spatial distribution information of the connection parts is generated. The harvesting scene is continuously captured in frames. The dynamic regions other than the harvested objects in adjacent frames are analyzed. The motion direction and velocity parameters of the dynamic regions are calculated. Dynamic interference sources that may affect the operation of the robotic arm are marked to form dynamic interference information of the growth environment. By integrating the morphological contour information, surface physical property visual representation information, spatial distribution information of connection parts, and dynamic interference information of the growth environment, a multi-dimensional visual perception set of the harvested object is formed. The robotic arm collaborative decision-making model, constructed based on the multi-dimensional visual perception set, analyzes the stress distribution characteristics of the connection points of the harvesting object through the collaborative decision-making model. Combined with the structural parameters of the end effector of the harvesting robotic arm, it determines the optimal contact area parameters of the harvesting object and the collaborative parameters of the robotic arm's posture, including: Spatial distribution information of connection parts and visual representation information of surface physical properties are extracted from the multi-dimensional visual perception set. The morphological contour information of connection parts is converted into a three-dimensional spatial coordinate sequence. Combined with the epidermal hardness level in the visual representation information of surface physical properties, a three-dimensional physical property mapping model of connection parts is constructed. Based on the aforementioned three-dimensional physical property mapping model, the stress distribution of the connection part under different contact forces is simulated using the finite element analysis method. The coordinates of the contact area corresponding to the minimum stress value are extracted as candidate contact area parameters. Obtain the structural parameters of the end effector of the harvesting robot arm, input the structural parameters into the three-dimensional physical property mapping model, and calculate the matching degree between the candidate contact area parameters and the end effector. The structural parameters include the clamping surface shape parameters, contact area parameters, and elastic coefficient parameters of the end effector. When the matching degree meets the preset requirements, the candidate contact area parameters are determined as the optimal contact area parameters; when the matching degree does not meet the preset requirements, the range of the candidate contact area parameters is adjusted, and the stress distribution and matching degree are recalculated until the optimal contact area parameters that meet the requirements are obtained. Dynamic interference information of the growth environment is extracted from the multi-dimensional visual perception set, the potential impact of interference sources on the movement trajectory of the robotic arm is analyzed, and the interference avoidance posture reference parameters of the robotic arm are determined. By combining the optimal contact area parameters and the interference avoidance posture reference parameters, the angle parameters of each joint of the robotic arm and the spatial posture parameters of the end effector are calculated, so that the end effector can avoid the interference source while maintaining a preset angle deviation from the optimal contact area, thus forming the robotic arm posture coordination parameters.

2. The method for operating a harvesting robotic arm based on visual feedback according to claim 1, characterized in that, The process of dynamically switching perspectives to capture images of the harvesting scene, adjusting the perspective parameters of the image acquisition device based on the proportion of the occluded area of ​​the harvesting object in the initial captured image, includes: Initial viewpoint images of the picking scene are acquired to obtain the initial image of the picking object. An image segmentation algorithm is used to separate the picking object area from the background area, and the occlusion area in the background area that occludes the picking object is marked. Calculate the proportion of the obstructed area in the picking target area to obtain the proportion of the obstructed area; When the proportion of the occluded area is less than or equal to a preset ratio, the initial view image is used as the initial visual image of the picking object; when the proportion of the occluded area is greater than the preset ratio, the view adjustment mechanism of the image acquisition device is activated. If the obstructed area is located below the object being picked, the image acquisition device is controlled to perform a viewing angle raising operation. Each time the viewing angle is raised by a preset angle, an image is acquired, and the proportion of the obstructed area in the newly acquired image is calculated until the proportion of the obstructed area is less than or equal to the preset proportion. If the obstructed area is located to the side of the object being picked, the image acquisition device is controlled to perform a side-shift operation. After each side-shift by a preset distance, an image is acquired, and the proportion of the obstructed area in the new image is calculated until the proportion of the obstructed area is less than or equal to the preset proportion. The image that finally meets the requirement of the proportion of the occluded area is determined as the initial visual image of the picking object, and the corresponding viewing angle parameters of the image acquisition device are recorded.

3. The method for operating a harvesting robotic arm based on visual feedback according to claim 1, characterized in that, Based on the three-dimensional physical property mapping model, the finite element analysis method is used to simulate the stress distribution at the connection point under different contact forces, and the coordinates of the contact area corresponding to the minimum stress value are extracted, including: The three-dimensional coordinate sequence of the connection part in the three-dimensional physical property mapping model is imported into the finite element analysis software, and the finite element mesh of the connection part is generated. Based on the surface hardness level in the visual characterization information of surface physical properties, a pre-established hardness level-material parameter mapping table is queried to assign corresponding elastic modulus and Poisson's ratio parameters to each element of the finite element mesh. The higher the hardness level, the greater the elastic modulus and the smaller the Poisson's ratio. Set multiple sets of different contact force parameters. Each set of contact force parameters includes the magnitude of the contact force and the direction of the contact force. The direction of the contact force covers multiple possible contact angles of the connection part. Finite element stress calculations were performed on each set of contact force parameters to obtain the stress distribution cloud map of the connection part under the action of the contact force, and the three-dimensional coordinates of the region with the minimum stress value in the stress distribution cloud map were extracted. Cluster analysis was performed on the coordinates of the stress minimum region corresponding to multiple sets of contact force parameters. Coordinates that are close to each other were grouped into the same candidate region, and the center coordinates of each candidate region were calculated. Based on the center coordinates of the candidate regions and the structural characteristics of the connection parts, the center coordinates of the stress-weak areas in the connection parts are selected as candidate contact region parameters.

4. The method for operating a harvesting robotic arm based on visual feedback according to claim 1, characterized in that, The step of generating a time-series motion instruction set for the robotic arm based on the optimal contact area parameters and the robotic arm posture coordination parameters includes: Based on the optimal contact area parameters, the contact start coordinates and contact end coordinates of the end effector are determined, the spatial distance between the contact start coordinates and contact end coordinates is calculated, and the joint motion speed limit in the robot arm posture coordination parameters is combined to plan the buffer motion trajectory of the contact phase. The speed of the buffer motion trajectory gradually decreases from the initial speed to the contact speed, forming the buffer motion trajectory information of the contact phase. Extract the skin hardness level corresponding to the optimal contact area from the surface physical characteristics visual representation information of the multi-dimensional visual perception set, determine the clamping force range of the end effector based on the skin hardness level, plan the force change curve of the clamping stage, the force gradually increases from the initial force at contact to the preset clamping force, and the rate of increase is negatively correlated with the skin hardness level, forming the force gradient change information of the clamping stage. Based on the spatial distribution information of the connection parts, the separation direction of the connection parts is analyzed. Combined with the angle parameters in the posture coordination parameters of the robotic arm, the initial angle and target angle of the separation stage are determined, the angle change and the required time are calculated, and the angle compensation motion trajectory of the separation stage is planned so that the end effector adjusts the angle along the separation direction of the connection parts during the separation process, thus forming the angle compensation motion information of the separation stage. The connection conditions between each stage of the time-sequenced action command are analyzed. When the end effector of the contact stage reaches the contact termination coordinate and the clamping force reaches the preset initial force, the clamping stage action command is triggered; when the clamping force reaches the preset clamping force and the connection part shows a tendency to separate, the separation stage action command is triggered, thus forming a stage connection rule. By integrating the buffer motion trajectory information of the contact phase, the force gradient change information of the clamping phase, the angle compensation motion information of the separation phase, and the phase connection rules, a time-sequenced action instruction set for the robotic arm is generated.

5. The method for operating a harvesting robotic arm based on visual feedback according to claim 4, characterized in that, The planned contact phase buffer motion trajectory, wherein the velocity of the buffer motion trajectory gradually decreases from the initial velocity to the contact velocity, includes: Determine the starting point coordinates and ending point coordinates of the motion during the contact phase. The starting point coordinates are the position coordinates of the end effector before the robotic arm performs the contact action, and the ending point coordinates are the contact start coordinates in the optimal contact area parameters. Calculate the straight-line distance between the coordinates of the starting point and the ending point of the motion, and combine it with the maximum motion speed of the robotic arm to determine the initial speed of the contact phase. The initial speed shall not exceed a preset proportion of the maximum motion speed of the robotic arm. Based on the surface hardness level in the visual representation information of the physical characteristics of the object being picked, and combined with the pre-calibrated hardness-maximum permissible contact speed relationship, the contact speed is determined. The lower the surface hardness level, the lower the contact speed. An exponential decay function is used to plan the velocity change curve. The velocity starts from the initial velocity and decays exponentially with the increase of the movement distance. When the end effector reaches the end point of the movement, the velocity decreases to the contact velocity. Based on the velocity change curve and the movement distance, the movement time of the contact phase is calculated, the movement time is divided into multiple time intervals, and the position coordinates and velocity parameters of the actuator at the end of each time interval are calculated. The position coordinates of each time interval are connected in chronological order to form the buffer motion trajectory of the contact phase. The position coordinates and corresponding velocity parameters of each point on the buffer motion trajectory are recorded as buffer motion trajectory information.

6. The method for operating a harvesting robotic arm based on visual feedback according to claim 1, characterized in that, The robotic arm controlling the harvesting process executes the time-series action instruction set, and during execution, it acquires a set of dynamic visual feedback from the harvesting scene in real time, including: The harvesting robotic arm is controlled to execute each stage of action according to the stage connection rules of the time-sequence action instruction set, and the image acquisition device is started simultaneously to acquire high-frequency images. The acquisition frequency is consistent with the stage change frequency of the robotic arm's action. Contact state recognition is performed on the high-frequency acquired images. The image difference algorithm is used to calculate the pixel changes in the contact area between the end effector and the picking object, determine the area change and positional offset of the contact area, and form contact state change information. The connection parts in the high-frequency acquired images are tracked and identified, the spatial coordinate changes of the connection parts in adjacent frames are extracted, the displacement distance and displacement direction of the picking object relative to the mother body are calculated, and the displacement trend information of the picking object relative to the mother body is formed. Trajectory tracking is performed on dynamic interference sources in high-frequency acquired images. The Kalman filter algorithm is used to predict the subsequent motion trajectory of the interference sources. Interference sources that may intersect with the motion trajectory of the robotic arm are marked to form motion trajectory information of environmental interference sources. By integrating the contact state change information, the displacement trend information of the picking object relative to the mother body, and the motion trajectory information of environmental interference sources, a dynamic visual feedback set for the picking scene is formed.

7. The method for operating a harvesting robotic arm based on visual feedback according to claim 1, characterized in that, The process of updating the input parameters of the robotic arm collaborative decision-making model based on the dynamic visual feedback set, and adjusting the stage connection parameters and motion intensity parameters of the time-series motion instruction set, includes: Contact state change information is extracted from the dynamic visual feedback set. When the position offset of the contact area exceeds a preset threshold, the position offset is converted into the posture correction parameter in the robotic arm collaborative decision model, and the input parameters of the robotic arm collaborative decision model are updated. Extract the displacement trend information of the picking object relative to the mother body from the dynamic visual feedback set, calculate the matching degree between the displacement trend and the separation stage action in the time-series action instruction set, and when the matching degree is lower than the preset value, input the displacement trend parameter into the collaborative decision model and recalculate the angle compensation parameter of the separation stage. Extract the motion trajectory information of environmental interference sources from the dynamic visual feedback set. When the distance between the interference source and the motion trajectory of the robotic arm is less than the safety threshold, input the speed and direction parameters of the interference source into the collaborative decision-making model and adjust the interference avoidance angle in the robotic arm posture collaborative parameters. Based on the updated collaborative decision-making model input parameters, the connection time difference of each stage of the time-sequence action instruction set is calculated. When the connection time between the contact stage and the clamping stage exceeds the preset range, the buffer movement time of the contact stage is shortened or extended, and the stage connection parameters are adjusted. Based on the corrected clamping force parameters output by the collaborative decision-making model, the force increase rate in the force gradient change information during the clamping stage is adjusted. When the object being picked shows a deformation trend, the force increase rate is reduced. Based on the corrected angle compensation parameters, the angle change rate during the separation stage is adjusted to ensure that the separation action is synchronized with the displacement trend of the object being picked. The adjusted stage connection parameters and motion intensity parameters are integrated into the time-series motion instruction set to generate an updated time-series motion instruction set, which controls the picking robotic arm to continue execution until the picking operation is completed.

8. A visual feedback-based harvesting robotic arm operating system, characterized in that, The visual feedback-based harvesting robotic arm operating system includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the visual feedback-based harvesting robotic arm operation method according to any one of claims 1-7.

Images