Target fruit dynamic tracking method and system based on visual perception and deep learning

Abstract

The application relates to the technical field of target tracking, and discloses a target fruit dynamic tracking method and system based on visual perception and deep learning. The method comprises the following steps: collecting original image data, performing image enhancement on the original image data through an encoder-decoder architecture self-adaptive regulation and control model, performing deep learning target detection on the large-view-area image after enhancement to extract fruit position information, guiding a small-view-area sensor to a picking preparation point, obtaining fruit occlusion rate and growth posture information through a multi-target segmentation network, combining depth information to perform dynamic tracking to obtain motion trajectory prediction data, calculating picking suitability scores and optimal picking points according to the occlusion rate, the growth posture and the motion trajectory, and generating picking execution instructions. The application improves fruit detection coverage and positioning accuracy in a complex orchard environment, and overcomes the limitation that picking points are single in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, in particular to a target fruit dynamic tracking method and system based on visual perception and deep learning. BACKGROUND

[0002] In the field of fruit picking, robot automatic recognition and picking technology has attracted widespread attention, but the complexity of orchard environment brings great challenges to automatic picking. The existing fruit picking robot system generally relies on a single visual sensor and traditional image processing algorithm. This design can achieve basic target recognition under ideal conditions, but has obvious limitations in actual orchard applications.

[0003] Light changes, shadow interference, fruit being blocked by branches and leaves, and fruit swinging caused by wind disturbance in the orchard environment make it difficult for a single visual system to simultaneously consider macroscopic perception and microscopic positioning. Especially in strong light or weak light conditions, traditional image processing methods are prone to overexposure or underexposure, resulting in loss of fruit target features; and in the case of partially blocked fruit, existing target detection algorithms often fail to accurately identify the complete shape and growth posture of the fruit, thereby affecting the accuracy of picking decisions. SUMMARY

[0004] The present application provides a target fruit dynamic tracking method and system based on visual perception and deep learning, which improves fruit detection coverage and positioning accuracy in complex orchard environments and overcomes the limitations of single picking point selection in existing technologies.

[0005] In a first aspect, the present application provides a target fruit dynamic tracking method based on visual perception and deep learning, which comprises:

[0006] Collecting raw image data and performing image enhancement through an adaptive regulation model of an encoder-decoder architecture to obtain an enhanced large field of view image;

[0007] Performing deep learning target detection on the enhanced large field of view image, extracting the position information of the fruit target and the picking preparation point position, and guiding the pre-set small field of view sensor to the picking preparation point position;

[0008] Collecting local image data and performing pixel-level segmentation through a multi-target segmentation network to obtain the occlusion rate and growth posture information of the fruit target;

[0009] According to the obtained depth information and the segmentation results of the small field of view sensor, performing dynamic tracking on the fruit target to obtain motion trajectory prediction data;

[0010] According to the occlusion rate, the growth posture information and the motion trajectory prediction data, a picking suitability score and an optimal picking point are calculated, and a picking execution instruction is generated.

[0011] In a second aspect, the present application provides a target fruit dynamic tracking system based on visual perception and deep learning, comprising:

[0012] An image enhancement module is configured to collect original image data and perform image enhancement through an adaptive regulation model of an encoder-decoder architecture to obtain large-view enhanced images;

[0013] A target detection module is configured to perform deep learning target detection on the large-view enhanced images, extract position information of fruit targets and picking preparation points, and guide preset small-view sensors to the picking preparation points;

[0014] A pixel-level segmentation module is configured to collect local image data and perform pixel-level segmentation through a multi-target segmentation network to obtain occlusion rates and growth posture information of fruit targets;

[0015] A dynamic tracking module is configured to perform dynamic tracking on the fruit targets according to obtained depth information and segmentation results of the small-view sensors to obtain motion trajectory prediction data;

[0016] A calculation module is configured to calculate a picking suitability score and an optimal picking point according to the occlusion rate, the growth posture information and the motion trajectory prediction data, and generate a picking execution instruction.

[0017] In the technical solution provided by the application, through the cooperative operation of the large-view sensor and the small-view sensor, the technical bottleneck that a single visual sensor cannot simultaneously consider global perception and local accurate identification is solved; multi-scale, multi-angle and multi-resolution cooperative perception is realized, and the fruit detection coverage and positioning accuracy in a complex orchard environment are significantly improved. The adaptive image regulation model with an encoder-decoder architecture effectively solves the influence of interference factors such as strong light, shadow, occlusion and motion blur in the orchard environment on image quality; through two special processing branches, pixel-level enhancement and global ISP parameter learning are respectively performed, so that the system can adapt to changing light conditions. The target detection and multi-target segmentation network based on deep learning enhances the feature extraction capability of the fruit; the multi-target instance segmentation network with the MASNet architecture realizes effective identification of the semi-occluded fruit, and provides accurate occlusion rate and fruit posture information for subsequent picking decision. The pseudo point cloud data and BEV view are constructed based on the RGB-D information, and the fruit positioning in the three-dimensional space is realized; the Kalman filtering algorithm with an adaptive process noise adjustment mechanism can accurately predict the motion trajectory of the fruit under wind disturbance, solving the problem of inaccurate tracking of dynamic targets in the traditional method. By comprehensively considering multiple key factors such as the occlusion rate of the fruit, the growth posture, the depth value, the motion state and the complexity of the surrounding environment, a scientific picking suitability scoring function is constructed; according to different picking difficulties, the working parameters of the end effector are automatically adjusted, and the limitation of single picking point selection in the prior art is overcome. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor based on these drawings.

[0019] Figure 1 An embodiment schematic diagram of the target fruit dynamic tracking method based on visual perception and deep learning in the embodiment of the application;

[0020] Figure 2 An embodiment schematic diagram of the target fruit dynamic tracking system based on visual perception and deep learning in the embodiment of the application. DETAILED DESCRIPTION

[0021] The embodiment of the present application provides a target fruit dynamic tracking method and system based on visual perception and deep learning. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" or "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0022] For ease of understanding, the specific flow of the embodiment of the present application is described below. Please refer to Figure 1 One embodiment of the target fruit dynamic tracking method based on visual perception and deep learning in the embodiment of the present application comprises the following steps:

[0023] Step S101, collect original image data, and perform image enhancement through an adaptive regulation model of an encoder-decoder architecture to obtain a large field of view image after enhancement;

[0024] Specifically, the original image acquisition of the orchard environment is performed by using a large-view sensor in a cascaded vision system, which continuously acquires image data of a large range of scenes at a resolution of 1920*1080 and a frequency of 30 frames per second within a set wide-angle field of view. The original image data is subjected to raster division processing, and the entire image is divided into a plurality of sub-region image blocks according to a fixed size (such as 160*160 pixels), and the overlap rate between the sub-regions is set to 20%, which effectively deals with the local perception distortion problem caused by local changes in light intensity, shadow interference and uneven imaging in the image. The set of image sub-regions obtained by division is sequentially input into an image self-adaptive regulation model of an encoder-decoder structure, which can realize a parallel and collaborative processing strategy of local enhancement and global unified adjustment based on the perception of the quality changes of different region images. In the encoder part of the model, a 5-layer continuous convolution structure is used to extract multi-level features from the input image sub-regions, each layer of convolution operation uses a convolution kernel with a size of 3*3 and a step of 2, and the number of channels is 64, 128, 256, 512 and 512 respectively, and batch normalization and activation function are used to ensure the stability and non-linear expression ability of feature extraction; the encoding process compresses the original image into a low-dimensional feature space representation layer by layer, and extracts the discriminative high-level semantic features in the image. After completing the feature encoding, the obtained encoded features are input into the decoder part for restoration processing, and the decoder also consists of 5 layers, but uses transposed convolution to realize step-by-step upsampling of the feature map and image reconstruction. Each layer of transposed convolution correspondingly maps the encoded low-dimensional features to a high-resolution space, restores the size structure of the original image, and introduces a skip connection of the encoding layer in the decoding process to retain the low-level detail feature information, so that the initial decoding result has higher structure restoration ability and texture fidelity. The initial decoding result is input into two independent processing branches for parallel processing, wherein the first processing branch focuses on pixel-level image enhancement, and internally connects three pixel enhancement modules in series, each module uses a residual structure design and introduces a dilated convolution operation to expand the receptive field range, while maintaining the integrity of the image details, enhancing the expression ability of blurred edges, weak contrast regions and occluded regions, and finally outputting a set of pixel-level enhancement features; at the same time, the second processing branch performs the learning task of image signal processing parameters, obtains the global light and color information of the input image through an attention mechanism module, and generates a set of global adjustment parameters including a color matrix and a Gamma correction value, which are used to express the overall regulation requirements of the entire image in terms of color distribution, brightness balance, tone curve, etc. The pixel-level enhancement features output by the first processing branch and the global adjustment parameters output by the second processing branch are fused, and the fusion operation includes weighted superposition and reverse mapping adjustment, so that the image not only retains the clear details after local enhancement, but also has overall consistency of uniform color and light characteristics, and obtains the enhanced large-view image.

[0025] In step S102, the enhanced large-view image is subjected to deep learning target detection to extract the position information of the fruit target and the picking preparation point position, and the preset small-view sensor is guided to the picking preparation point position.

[0026] Specifically, the enhanced large-view image is input into a target detection network for feature extraction processing. The network is designed with a lightweight structure, and the main part of the network is constructed by using deep separable convolution. The deep separable convolution reduces the computational complexity and parameter quantity of the network by decomposing the standard convolution operation into two stages of channel-by-channel convolution and point-by-point convolution, while maintaining the sensitivity to the spatial structure and channel features of the image. After processing the enhanced image under this structure, an initial feature map that retains the semantic information of the fruits is obtained. The initial feature map is input into a CBAM module integrated with attention mechanisms for joint modeling of channel attention and spatial attention. In the channel attention mechanism, the network performs average pooling and maximum pooling operations on the feature map along the spatial dimension, and forms a channel weight vector through a fully connected layer, thereby highlighting the key channel features related to fruit recognition. In the spatial attention mechanism, the feature map is pooled along the channel dimension, and a spatial response weight map is obtained by combining the convolution operation, which emphasizes the edge profile, boundary and high response area in the image. Through the cascade fusion processing of the two attention mechanisms, the enhanced feature map has stronger discriminability and positioning ability in terms of representation ability. The enhanced feature map is input into three detection heads of different scales for target detection processing. Each detection head identifies target features within a specific scale range, and the multi-scale fusion detection mechanism ensures consistent performance when identifying small, occluded, distant and close-range fruits. Each detection head outputs an information set containing fruit category probability, bounding box coordinates and confidence score. By aggregating the multi-scale detection results, the position distribution of the target fruit in the image plane is obtained. In order to eliminate overlapping boxes and redundant recognition, non-maximum suppression processing is performed on the detection results. The bounding box with the highest confidence score and low overlap with other candidate boxes is selected as the final output result, generating a set of fruit target bounding boxes with accurate boundaries and no redundant overlap. According to the fruit target bounding box information, the center point coordinates of each fruit bounding box are calculated. The center point serves as the picking preparation point of the fruit, and the depth value corresponding to the point is extracted from the depth channel of the large-view sensor. Thus, the two-dimensional image center point and spatial depth information are combined to represent the position of the picking preparation point in three-dimensional space. After obtaining the spatial position, a series of coordinate transformation operations are performed to map the point from the perception layer to the control layer. Through the rigid transformation matrix previously established between the large-view sensor and the robot reference coordinate system, the picking preparation point is converted from the image space to the robot body reference coordinate system, and then further converted to the motion control coordinate system of the picking mechanism through the transformation matrix defined by the robot structure parameters, thereby completing the coordinate chain mapping from the perception point to the execution point, and generating a set of pose control parameters for driving the small-view sensor to move accurately. Based on the above pose control parameters, the robot motion control unit guides the small-view sensor to move quickly and accurately to the specified picking preparation point position through interpolation calculation and real-time feedback, ensuring that the small-view sensor's field of view contains the target fruit area.

[0027] Step S103, collect local image data and perform pixel-level segmentation through a multi-target segmentation network to obtain the occlusion rate and growth posture information of the fruit target;

[0028] Specifically, the picking preparation point provided by the large-view visual module in the early stage guides the small-view sensor to move to the target area, and high-resolution local image data is collected in this view range. The image contains complex details such as fruit bodies, leaf and branch occlusion structures, fruit stems, and adjacent background textures. Then, the image is input into the multi-target segmentation network for initial processing. The network outputs an initial pixel-level mask image according to the pixel-level recognition mechanism. The mask semantically divides the fruit, leaves, and background on the image level, and labels the initial boundary between the possible fruit area and the interference area. In order to improve the accuracy of boundary determination and the hierarchical robustness of segmentation, ResNext-101 with strong feature extraction capability is used as the backbone structure, and a feature pyramid network is combined for multi-scale feature extraction, so that each type of object in the image can be effectively expressed at different spatial scales, generating a feature map set with high resolution and multi-level context information. On the basis of the multi-scale feature map set, boundary box regression and classification operations are performed, that is, the network regression module determines the boundary box position of each candidate target, and the classification module verifies whether it is a fruit category, forming a set of accurate boundary box information after discrimination. Based on the area defined by the boundary box, the system activates the mask generation branch of the segmentation network again, and uses local feature up-sampling, jump connection, and convolution fusion to construct a refined mask image. The mask can accurately restore the complete structure of the fruit at the pixel level and effectively eliminate the interference caused by edge blur and partial occlusion, obtaining an accurate fruit segmentation mask image. Based on the accurate segmentation mask, the occlusion rate of the fruit target is calculated. By calculating the area difference between the actual visible area and the overall predicted area in the mask, it is identified whether the current fruit has significant occlusion at this view angle. The higher the occlusion rate, the more difficult it is to identify and safely pick the fruit. At the same time, the edge contour information of the accurate mask is extracted and a boundary point set is constructed. The directional modeling is performed in combination with the image coordinate structure, and the angle between the principal direction vector and the vertical direction of the image is calculated to obtain the angle between the current growth posture of the fruit and the gravity direction. This angle information represents the inclination degree, growth direction of the fruit, and whether it is suitable for picking from the current angle. The two types of structured information, occlusion rate and growth posture, are fused into the system picking decision module to participate in the scoring and priority sorting of the fruit pickability.

[0029] Step S104, according to the obtained depth information and the segmentation result of the small-view sensor, performing dynamic tracking on the fruit target to obtain motion trajectory prediction data;

[0030] Specifically, the image data containing depth channel is collected by using a small field of view sensor, that is, an RGB-D image is obtained, wherein each pixel point carries corresponding depth value while having color information. According to the intrinsic matrix of the camera, a coordinate back-projection operation is performed on each pixel point in the RGB-D image, which is converted from two-dimensional image coordinates to three-dimensional space coordinates, and a pseudo point cloud data structure containing spatial position and color information is constructed. Based on the pseudo point cloud data of the fruit region, a horizontal XOY coordinate system is established with the picking mechanism as the origin, the coordinate system is projected on the horizontal plane, the position of the fruit in the three-dimensional coordinate system is mapped to the two-dimensional bird's eye view plane to form a BEV (bird's eye view) coordinate expression mode, which is more suitable for motion trajectory calculation and path fitting in dynamic tracking. At the same time, the center point coordinates of the fruit boundary box and the corresponding depth value are extracted from the segmentation result of the small field of view sensor. By analyzing the center position of the boundary box inside the segmentation mask and combining the depth information of the point, the three-dimensional coordinate expression of the fruit in the local field of view is obtained, and the data is associated with the constructed BEV spatial position to obtain the three-dimensional spatial position of the fruit target. The three-dimensional spatial position of the fruit target is matched between frames, the difference in spatial position of the fruit in consecutive frames is compared, the trajectory path of the fruit in the motion process is recognized, and the complete motion trajectory is constructed combined with the time sequence data. The Kalman filter algorithm is introduced to estimate the dynamic state of the trajectory, which takes the spatial position and velocity state of the fruit as internal variables, and combines the time evolution law to predict and update between each frame. By combining the observation value and the predicted state, a smoother and more continuous fruit motion trajectory curve is generated, and the fruit state prediction data is obtained. Based on the fruit state prediction data, an adaptive process noise adjustment mechanism is introduced, the amplitude of the fruit position change between adjacent image frames is monitored, and the process noise covariance parameter in the filter is dynamically adjusted, that is, when the fruit moves violently and has large displacement in consecutive frames, the response ability of the system to uncertainty is improved, and the fault tolerance range of the prediction area is increased; when the fruit moves slowly or tends to be stable, the covariance value is reduced to enhance the prediction accuracy. The fruit motion trajectory prediction data is output based on the above state estimation and dynamic adjustment model.

[0031] In step S105, the picking suitability score and the optimal picking point are calculated according to the occlusion rate, the growth posture information and the motion trajectory prediction data, and a picking execution instruction is generated.

[0032] Specifically, the multi-source fruit state information output by the previous modules is comprehensively processed and integrated to construct a structured fruit picking factor set, which includes the fruit occlusion rate calculated from the pixel-level segmentation result, the growth posture angle obtained from the edge morphology analysis, the fruit depth value and motion state output by the trajectory tracking module, and the surrounding environment complexity index calculated by the environment perception unit. These factors collectively describe the pickability, picking difficulty, and picking risk of the fruit in the current environment and time. Based on this factor set, a multi-dimensional picking suitability scoring function is constructed. The scoring function converts each factor into a standardized index value through weighting or model mapping and performs comprehensive calculation to generate a picking suitability score value, which reflects whether the target fruit has a high picking priority under the current conditions. All identified fruits are sorted by suitability score, and a score threshold is set to filter out the fruit set suitable for immediate picking as the list of fruits to be picked. For each fruit to be picked, a detailed structure analysis process is started, and fruit edge features and internal contours are extracted based on high-precision segmentation masks in small field images. The fruit stem connection area is identified, and the fruit stem connection point is located in the narrow area where the upper or side of the fruit contacts the branch. This point is considered an important reference point for the contact between the picking clamp and the target. After identifying the fruit stem connection point, a local coordinate system is established on the fruit surface, and according to the spatial relationship between the point position and the geometric center of the fruit, the optimal approach path and the optimal picking point position are calculated to enable the picking tool to achieve stable clamping with the shortest path and smallest interference angle. According to the optimal picking point, the dynamic state of the fruit reflected by the picking suitability score is used to customize the execution strategy, such as assigning a high-speed, low-damping mode for high-score target fruits to achieve fast picking, and using a medium-speed, stable control strategy for boundary-value fruits to balance safety and efficiency. The spatial position, clamping posture, approach direction, and control mode parameters of the optimal picking point are packaged into a standardized picking execution instruction, which is sent to the end effector through the control link, and the motion trajectory and clamping strategy are adjusted in real time during the picking process based on feedback.

[0033] In the embodiment of the present application, through the cooperative operation of the large field of view sensor and the small field of view sensor, the technical bottleneck that a single visual sensor cannot simultaneously consider global perception and local accurate identification is solved; multi-scale, multi-angle and multi-resolution cooperative perception is realized, and the fruit detection coverage and positioning accuracy in a complex orchard environment are significantly improved. The adaptive image regulation model based on the encoder-decoder architecture effectively solves the influence of interference factors such as strong light and shadow, occlusion and motion blur in the orchard environment on the image quality; through two special processing branches, pixel-level enhancement and global ISP parameter learning are respectively performed, so that the system can adapt to variable light conditions. The target detection and multi-target segmentation network based on deep learning enhances the feature extraction capability of the fruit; the multi-target instance segmentation network of the MASNet architecture realizes effective identification of the semi-occluded fruit, and provides accurate occlusion rate and fruit posture information for subsequent picking decision. The pseudo point cloud data and BEV view are constructed based on the RGB-D information, and the fruit positioning in the three-dimensional space is realized; the Kalman filtering algorithm with an adaptive process noise adjustment mechanism can accurately predict the motion trajectory of the fruit under wind disturbance, solving the problem of inaccurate tracking of dynamic targets in the traditional method. By comprehensively considering multiple key factors such as the occlusion rate of the fruit, the growth posture, the depth value, the motion state and the complexity of the surrounding environment, a scientific picking suitability scoring function is constructed; according to different picking difficulties, the working parameters of the end effector are automatically adjusted, overcoming the limitation of single picking point selection in the prior art.

[0034] In a specific embodiment, before step S101 is performed, the following steps are further included:

[0035] The parameters of the large field of view sensor are set so that the field of view angle of the large field of view sensor is greater than 120°, the resolution reaches 1920x1080 pixels, and the acquisition frequency is set to 30 frames / second; and the parameters of the small field of view sensor are set so that the field of view angle of the small field of view sensor is controlled within the range of 30°-60°, the resolution reaches 2560x1440 pixels, and the acquisition frequency is set to 60 frames / second;

[0036] The large field of view sensor and the small field of view sensor are rigidly connected and installed, and the precise positional relationship between the sensors and the mechanical structure is determined through a hand-eye calibration algorithm to obtain the installation positions of the large field of view sensor and the small field of view sensor;

[0037] Based on the installation positions of the large field of view sensor and the small field of view sensor, a robot body base coordinate system, a robot picking mechanism coordinate system, a large field of view sensor coordinate system, a large field of view sensor target picking preparation point coordinate system, a small field of view sensor coordinate system and a small field of view sensor target picking point coordinate system are established to obtain a cascaded vision system.

[0038] Specifically, according to the spatial range of the orchard working environment and the visual coverage demand of the picking task, a large field of view sensor suitable for macroscopic perception is selected. The sensor needs to have a wide viewing angle and medium resolution to complete the initial detection and positioning of all fruits in the scene. In terms of specific parameter configuration, the field of view angle is set to be greater than 120° to cover a larger orchard area at one time without frequent movement of the viewing angle. At the same time, the resolution is set to 1920×1080 pixels to ensure image clarity and control data volume for real-time processing. The acquisition frequency is set to 30 frames per second to ensure that the number of image frames is sufficient to support continuous tracking of fruits during low-speed movement. The small field of view sensor used in conjunction with it focuses on local high-precision imaging of a single fruit area. Its design principle is to highlight resolution and timeliness, so the field of view angle is controlled within the range of 30° to 60° to narrow the shooting range and enhance the target focusing effect. At the same time, the resolution is set to 2560×1440 pixels to enable the system to accurately obtain detailed information such as fruit edges and stem connections. The acquisition frequency is increased to 60 frames per second to achieve stable capture of high-speed shaking targets and support subsequent pixel-level mask segmentation and pose extraction operations. After completing the sensor parameter setting, the large and small field of view sensors are installed in a rigid structure on the same end module of the picking robot arm. This structure connection has the physical characteristics of anti-torsion, anti-vibration, and no relative displacement, ensuring that the spatial relationship between the two sensors remains fixed during operation, avoiding visual mismatch caused by shaking or external forces. After installation, the spatial transformation relationship between the sensors and the robot mechanical structure is obtained through a hand-eye calibration algorithm. This process uses Zhang's calibration method combined with a nonlinear least squares optimization algorithm for calculation. A calibration board containing high-contrast markers is set up, and at least twenty image sequences are taken at multiple angles and distances. The corner pixel coordinates are extracted through image processing algorithms, and combined with the real-time pose data of the picking mechanism, a mapping function from camera image coordinates to mechanical coordinates is established. During the optimization process, the system calculates the rotation vector and translation vector and combines them into a rigid transformation matrix. The re-projection error is used as a calibration accuracy indicator for adjustment, obtaining the accurate three-dimensional installation position and attitude direction parameters of the two sensors in the robot coordinate system. In order to realize the cooperation between multiple visual sources and the complete mapping from visual perception to motion control, the system establishes a standardized multi-coordinate system structure after obtaining the position parameters. The coordinate system takes the robot body base coordinate system as the global reference frame, and on this basis, six sub-coordinate systems are sequentially constructed: the robot picking mechanism coordinate system, the large field of view sensor coordinate system, the large field of view target picking preparation point coordinate system, the small field of view sensor coordinate system, and the small field of view target picking point coordinate system. These coordinate systems are connected through homogeneous transformation relationship.The robot body coordinate system is used to describe the position and posture of the whole picking platform, the robot picking mechanism coordinate system is used to describe the geometric action structure of the mechanical arm and the end effector, the large field of view sensor coordinate system corresponds to the imaging position and direction thereof, the large field of view target picking preparation point coordinate system is used to represent the spatial position of the fruit after macroscopic preliminary positioning, and the small field of view sensor coordinate system is used to represent the angle and coordinate range of local image acquisition. The transformation relationship between the coordinate systems is encapsulated by a rigid body rotation matrix and a translation vector, and a coordinate mapping chain is constructed. The preliminary position of the fruit obtained by the large field of view is first converted from the large field of view sensor coordinate system to the robot body coordinate system, and then translated to the picking mechanism coordinate system by the robot control system, so as to guide the small field of view sensor to move to the picking preparation point view angle range. The target picking point obtained by the small field of view is converted from the small field of view coordinate system to the robot execution layer coordinate system in the reverse direction, so as to control the clamping trajectory and clamping angle posture of the end effector gripper. In the whole cascaded vision system, the large field of view provides large-range fruit retrieval and distribution mapping function, and the small field of view is responsible for fine identification and dynamic tracking. The two are interconnected through the coordinate system, and accurate rigid installation and calibration technology is relied on to realize unified expression of spatial information and time sequence consistent control, thereby forming a high-robustness multi-level vision fusion system which can stably operate in an orchard environment.

[0039] In a specific embodiment, the process of performing step S101 can specifically include the following steps:

[0040] Raw image data is collected by the large field of view sensor in the cascaded vision system, and the raw image data is subjected to grid division processing to obtain a set of divided image sub-regions;

[0041] The set of divided image sub-regions is input into an adaptive regulation and control model of an encoder-decoder architecture, and feature encoding is performed on the image coding features through a 5-layer convolution structure in the encoder to obtain image coding features;

[0042] Feature decoding is performed on the image coding features through a 5-layer transposed convolution structure in the decoder to obtain an initial decoding result;

[0043] The initial decoding result is input into a first processing branch to perform pixel-level enhancement to obtain pixel-level enhancement features, and the initial decoding result is input into a second processing branch to perform ISP parameter learning to obtain global adjustment parameters;

[0044] The pixel-level enhancement features and the global adjustment parameters are fused to obtain an enhanced large field of view image.

[0045] Specifically, the original image data is collected by a large field of view sensor in a cascade vision system, which covers a wide range of images and can capture multiple fruit targets and their natural background environment at once, including direct sunlight, leaf shading, fruit branch intersection and other typical interference elements. To improve the adaptability of the local area, the collected original image is rasterized and divided according to the preset rules, and the entire image is equally divided into a set of image sub-regions with consistent size and certain overlapping areas. Each sub-region covers several local segments of fruit and their environment background. The overlapping area can effectively reduce the fragmentation of boundary information and provide sufficient context cross features for the network model. The image sub-region set after division is input into the image adaptive regulation model based on the encoder-decoder architecture. In the encoder part of the model, a 5-layer convolution structure is used to perform feature encoding operation on each image sub-region. The 5-layer convolution structure uses a standard convolution kernel size, usually 3x3, and is attached with batch normalization and nonlinear activation function after each layer to maintain stability and nonlinear expression ability during feature extraction. The number of channels increases from the shallow layer to the deep layer, allowing the network to gradually compress the image spatial dimension while enhancing the semantic feature perception ability, and extracting intermediate feature maps containing information such as light changes, edge structures, and color distribution. The output of the encoder is the image encoding feature. The image encoding feature is decoded by a 5-layer transposed convolution structure in the decoder, which gradually restores the spatial size of the image and reconstructs the intermediate image representation with the original image structure perception ability. Each transposed convolution layer is connected with a jump connection mechanism, which allows the spatial detail information of the shallow layer to be fused with the semantic information of the deep layer in the same feature scale, maximizing the preservation of image original texture, edge and contour details. After 5 layers of decoding, an initial decoding image result is obtained, which maintains the integrity of the spatial structure. The initial decoding image is then processed in two parallel processing structures for pixel-level enhancement and global parameter learning. The first processing branch focuses on performing pixel-level enhancement of the image, and three pixel enhancement modules are set in series inside the branch. Each module contains a residual structure, a dilated convolution and an attention mechanism to highlight the recovery ability of the image in the case of boundary blur, dark area and texture loss. The residual structure helps the model to learn the increment of image changes, the dilated convolution expands the receptive field to capture long-range information, and the attention mechanism adaptively adjusts the enhancement intensity of each region, so that the real fruit region in the image obtains higher response weight. After processing, the first branch outputs the pixel-level enhanced image feature map. At the same time, the second processing branch learns the ISP parameters of the entire image, which is the adaptive estimation of the image signal processing parameters, mainly including brightness balance, color correction and Gamma adjustment.The branch internally constructs a lightweight fully connected network structure, and the input is the global feature map of the initial decoding image. A set of matrix coefficients for controlling image color conversion and brightness adjustment function curve parameters are calculated through a set of attention mapping, forming a set of global image adjustment parameters. The goal of this processing path is to unify the color style and contrast range of the entire image, ensuring that the image under different regions and different lighting has consistent visual characteristics, providing a more stable input distribution for the subsequent detector. After the two branches complete processing, the pixel-level enhancement features and global adjustment parameters are fused, including weighted integration in the image space and mapping inverse calculation in the feature space, and finally an enhanced large view image is output.

[0046] In a specific embodiment, the process of performing step S102 can specifically include the following steps:

[0047] The enhanced large view image is input into the target detection network for depth separable convolution processing to obtain an initial feature map.

[0048] The initial feature map is calculated by the CBAM attention module for channel attention and spatial attention to obtain an enhanced feature map.

[0049] The enhanced feature map is detected by three different scale detection heads for multi-scale feature fusion detection to obtain the position information of the fruit target.

[0050] The position information is subjected to non-maximum suppression processing to obtain the accurate fruit target bounding box.

[0051] The center point coordinates of each fruit target are calculated based on the accurate fruit target bounding box as the picking preparation point coordinates, and the depth values of the corresponding points are obtained from the large view sensor to obtain the picking preparation point position.

[0052] The coordinate transformation matrix is performed on the picking preparation point position to convert the picking preparation point position from the large view sensor coordinate system to the robot body base coordinate system, and then to the robot picking mechanism coordinate system to obtain the position parameters for controlling the movement of the small view sensor.

[0053] Based on the position parameters for controlling the movement of the small view sensor, the small view sensor is guided to the picking preparation point position.

[0054] Specifically, the enhanced large-view image is input into a target detection network for feature extraction and spatial positioning processing. The target detection network adopts a depth separable convolution structure as the backbone network. The depth separable convolution separates the standard convolution into two stages of channel-by-channel convolution and point-by-point convolution, respectively captures the channel internal features and cross-channel interaction, improves the model's recognition ability of the fruit outline, color boundary and background structure in the image, and obtains an initial feature map after processing through this module. The feature map retains the information distribution and outline pattern of the key regions in the image. Then, the CBAM attention module is used to calculate the channel attention and spatial attention of the initial feature map. In the channel attention stage, global description vectors are extracted by performing average pooling and maximum pooling operations on the spatial dimensions of the initial feature map, and channel weighting coefficients are generated through a multilayer perceptron, so as to enhance the response of the fruit channel and suppress irrelevant background features. In the spatial attention stage, the spatial weight map is generated by performing pooling and convolution processing on the channel dimension, so that the network focuses on the regions with high response degree in the image, such as fruit edge, fruit stem connection and color mutation. The enhanced feature map is output through the CBAM module. Through three detection heads of different scales, multi-scale feature fusion detection is performed on the enhanced feature map. These detection heads correspond to the receptive regions of small, medium and large size targets respectively, and can comprehensively cover various sizes and scale levels of fruits appearing in the image. The detection head outputs the class probability, boundary box position and confidence information of the fruit through convolution, normalization and activation operations. Each detection head processes a group of down-sampled feature maps and extracts candidate targets in a specific scale range, so that the system can maintain high recognition consistency and coverage when facing orchard images with varying fruit sizes and uneven distribution. In order to remove the repeated candidate boundary boxes generated in the multi-scale detection process, a non-maximum suppression processing mechanism is introduced based on the target detection result. The mechanism sorts the confidence of the candidate boxes of the same class and high overlap degree, and sequentially removes the low confidence boundary boxes with overlap exceeding the set threshold, so as to retain the unique and optimal boundary box of each fruit, realize the unique determination of the boundary and the accurate extraction of the edge, and output a set of fruit target boundary box data set with clear structure and accurate coverage. The center point coordinates of the fruit target are calculated based on each accurate boundary box. The center point is defined as the picking preparation point of the fruit, which serves as the geometric reference for subsequent guided fine positioning. In order to map the center point from the image coordinate system to the real world coordinate system, the depth value corresponding to the center point is extracted from the large-view image, and the physical position of the point in the three-dimensional space is recovered by combining the imaging geometric relationship. This three-dimensional coordinate is the spatial picking preparation point position of the fruit at the current time and current view, which contains the relative coordinate relationship in the horizontal, vertical and front-back directions.The three-dimensional picking preparation point position is sequentially subjected to multi-stage coordinate transformation operation to complete the conversion from the perception space to the execution space. The picking point is mapped from the image coordinates to the robot reference coordinate system through the transformation matrix from the large field of view sensor to the robot body base coordinate system obtained by pre-calibration, and then is transferred from the reference coordinate system to the working coordinate system of the picking mechanism itself through the transformation matrix defined in the structure parameters of the picking mechanism, to obtain the pose control parameters for controlling the joint movement of the picking end and the small field of view sensor. The control parameters include the target position, and also include the expected reaching pose and the approaching angle and other motion planning information, to ensure that the subsequent movement path is smooth and the clamping pose is reasonable. The pose control parameters are sent to the motion control unit of the picking mechanism, and the small field of view sensor is guided to move accurately to the target picking preparation point position along the planned trajectory. In this process, closed-loop correction is performed in combination with the end position feedback signal to improve the positioning accuracy and complete the focusing and fixed angle adjustment of the small field of view sensor, so that the final field of view coverage area completely contains the target fruit area and the surrounding structure.

[0055] In a specific embodiment, the process of performing step S103 can specifically include the following steps:

[0056] Local image data is collected by the small field of view sensor in the cascade vision system, and the local image data is input into the multi-target segmentation network for pixel-level mask segmentation to obtain an initial pixel-level mask.

[0057] A ResNext-101-FPN structure is used to extract multi-scale feature maps from the initial pixel-level mask to obtain multi-scale feature representations.

[0058] Boundary box regression and classification calculation are performed on the multi-scale feature representations to obtain accurate boundary boxes of the fruit targets, and pixel-level mask generation is performed on the multi-scale feature representations based on the accurate boundary boxes of the fruit targets to obtain accurate segmentation masks of the fruit targets.

[0059] The occlusion rate of the fruit target is calculated based on the accurate segmentation mask, and the growth pose information of the fruit target is obtained by calculating the fruit pose angle and the included angle in the vertical direction based on the accurate segmentation mask.

[0060] Specifically, local image data is acquired using a small field-of-view sensor within a cascaded vision system. Due to its narrower field of view, the small field-of-view sensor can focus on the target fruit region with a more concentrated imaging angle, acquiring local image data containing details of the fruit structure, the position of the stem, and information on occlusion by surrounding leaves. This local image data is then input into a multi-target instance segmentation network to perform pixel-level mask segmentation. This segmentation network is designed to integrate instance detection and semantic segmentation capabilities, enabling it to identify the category information of different target objects in the image and distinguish instances of the same category. It generates a pixel-level binary mask map with the same scale as the original image within each target region, obtaining the initial mask result. This mask covers the main outline of the fruit, edge transition areas, and some occluded but predictable areas. To improve the adaptability and accuracy of the mask segmentation results on fruits of different scales and structural features, a ResNext-101 network is used as the backbone structure based on the initial mask result, combined with a feature pyramid network for multi-scale feature map extraction. ResNext-101 achieves parallel feature extraction within channels through grouped convolutions, significantly reducing computational complexity while maintaining high network expressive power. It also demonstrates excellent representation capabilities for complex structures on fruit surfaces, such as textures, color blocks, edges, and stem connections. Combined with an FPN structure, it extracts feature maps at multiple scale levels from the image, for example, corresponding to the original... Figure 1The feature maps of scaling ratios of 1 / 4, 1 / 8, 1 / 16, etc. retain more edge details and texture information in low-level feature maps, while high-level feature maps extract the overall semantic contour and spatial consistency of the fruit. Through transverse fusion and up-sampling mechanism, a set of fusion feature representation containing multi-scale semantic features is formed, and a set of multi-scale feature maps with stable structure and rich semantics is obtained. Based on the above multi-scale feature map representation, the boundary box regression and target classification operations in the target detection process are performed. The boundary box position, width and height size and target class of each candidate region are predicted through the convolution module, and the redundant candidate boxes are processed through non-maximum suppression to output accurate boundary box data closely fitted to the fruit position. Each boundary box defines a region of interest, and the system then enables the instance mask generation module inside the boundary box to perform pixel-level mask reconstruction based on the local feature map. This process uses a combination structure of multi-layer transposed convolution and skip connection, generates target instance masks consistent with the original image resolution through step-by-step up-sampling and fine decoding, and the mask boundary not only has clear spatial continuity, but also can be flexibly fitted in the transition zone between the fruit edge and the background, obtaining a fruit segmentation mask graph with complete structure and accurate boundary. Based on the accurate segmentation mask calculation, geometric and area analysis is performed. By counting the total number of pixels marked as fruit in the mask graph, and comparing the actual visible pixel area with the occluded or incomplete area, the visible proportion of the fruit under the current image perspective is evaluated. This proportion can be used to estimate the occlusion degree of the fruit, forming the occlusion rate index. The higher the occlusion rate, the more the fruit is occluded by surrounding leaves, branches or other fruits, and the more difficult it is to recognize and hold. A target with a low occlusion rate indicates a clear target boundary and complete structure, making it suitable as a priority picking object. At the same time, based on the edge coordinate points of the accurate mask graph, contour extraction is performed to construct a set of fruit shape boundary points, and the main direction of the shape is calculated. The principal component analysis operation is performed on the boundary point set to extract the first principal axis direction as the growth direction of the fruit, and the angle between the main direction vector and the vertical direction of the image is calculated as the fruit growth posture angle. The posture angle reflects the degree of inclination or growth direction deviation of the fruit, which is used to guide the approach path adjustment of the end effector and the posture control of the gripper in spatial control. The occlusion rate and growth posture angle are used as structured perception indicators to evaluate whether the fruit has good recognition clarity and picking operability. High occlusion rate or large posture angle will cause recognition failure or operation risk such as gripper detachment, so these two indicators are used as key inputs in the subsequent picking suitability scoring model to evaluate the picking priority of the target fruit.

[0061] In a specific embodiment, the process of performing step S104 can specifically include the following steps:

[0062] Based on the depth information, RGB-D information is obtained, and a depth value is assigned to each pixel point. The two-dimensional image coordinates are converted into three-dimensional space coordinates by combining the camera intrinsic matrix, and pseudo point cloud data of the fruit region is obtained.

[0063] Based on the pseudo point cloud data of the fruit region, a horizontal plane XOY coordinate system is established with the picking mechanism as the origin, and the three-dimensional position of the fruit is projected onto the horizontal plane to form a BEV coordinate representation.

[0064] The segmentation result of the small field of view sensor is extracted to obtain the fruit bounding box center point coordinates and the corresponding depth value, and the three-dimensional space position of the fruit target is calculated by combining the BEV coordinate representation.

[0065] The three-dimensional space position of the fruit target is matched between frames to obtain the motion trajectory of the fruit target, and a Kalman filter algorithm is used to predict the state of the motion trajectory of the fruit target to obtain fruit state prediction data.

[0066] Based on the fruit state prediction data, an adaptive process noise adjustment mechanism is introduced, and the process noise covariance matrix is dynamically adjusted according to the change in the fruit position between adjacent frames to obtain motion trajectory prediction data.

[0067] Specifically, the depth perception ability of the small field-of-view sensor in the cascaded vision system is relied on to obtain an RGB-D image containing color image information and spatial depth data, where each pixel point has a color value composed of RGB channels and is also attached with a corresponding depth value representing the distance between the pixel and the camera during imaging. In order to convert the fruit information in the two-dimensional image into a manipulable geometric entity in the three-dimensional space, the depth value is combined with the camera intrinsic matrix, and the focal length and principal point coordinates in the intrinsic matrix are used to perform back-projection processing on each pixel in the image, thereby realizing the mapping of two-dimensional image coordinates to three-dimensional world coordinates. In this process, each pixel point is assigned a spatial position, and in the fruit recognition area, a set of dense points with spatial depth attributes is formed, i.e., pseudo point cloud data of the fruit area. Based on the pseudo point cloud data of the fruit area, a horizontal plane XOY coordinate system parallel to the ground is established with the position of the end effector of the picking mechanism as the coordinate origin. In this coordinate system, the three-dimensional position of the fruit is converted to BEV representation through projection, which is a method of projecting three-dimensional points in the pseudo point cloud onto the XOY plane to obtain the planar position distribution of the fruit in the overhead view, so that the system can intuitively express the dynamic trajectory of the fruit on the ground coordinate, and facilitate operations such as trajectory fitting, motion trend judgment and path optimization. Based on this coordinate system, the relative displacement change of the fruit target in the BEV plane is continuously monitored in the subsequent time sequence, thereby constructing the planar path data set of the fruit changing with time. At the same time, after the small field-of-view sensor is guided to the target fruit picking preparation point, high-resolution imaging of the fruit is performed and pixel-level instance segmentation is completed. The system extracts the center point of the fruit's bounding box in the small field-of-view segmentation result as the target reference coordinate in the local image, and synchronously extracts the depth value corresponding to the center point from the depth channel of the sensor to obtain its three-dimensional position representation in the local coordinate system. The three-dimensional coordinate is geometrically corresponding to the BEV coordinate system to complete the information fusion of the two types of vision sensors in the spatial layer, so that the system has multi-resolution and multi-angle view support for spatial tracking of the fruit, thereby improving the tracking accuracy and stability. The three-dimensional spatial position of the fruit target is matched between frames, and by calculating the Euclidean distance, relative direction and speed change of the fruit position at consecutive times, it is judged whether the target in the current frame is the continuation instance of the target in the previous frame, and the consistency of each fruit in time sequence is maintained through the target ID tracking strategy. After successful matching, the motion trajectory line of the fruit in the three-dimensional space arranged in time sequence is constructed, reflecting the dynamic shaking behavior of the fruit in the natural environment due to wind disturbance, gravity swing or branch elasticity. In order to improve the robustness of fruit trajectory analysis, a Kalman filter algorithm is introduced to predict the state of the above motion trajectory, which models the current three-dimensional position and speed state of the fruit, predicts the fruit position in the current frame based on the state of the previous frame at each frame, and fuses it with the current observed position to generate a smooth and continuous state estimation result.The Kalman filtering process includes state transition equation and observation update equation. In the state transition, it is assumed that the fruits move in a small range with constant speed or slowly changing acceleration. In the observation update, the actually measured position in the current frame image is introduced. The state is optimally fused by using the Bayesian estimation theory to obtain a smooth path representation closer to the real trajectory, which is suitable for maintaining stable identification and tracking of fruits in frames where the fruits are partially blocked by leaves or the image is blurred. Considering that the motion law of fruits in a natural scene does not completely follow the linear system assumption, an adaptive process noise adjustment mechanism is introduced in the Kalman filtering framework. The mechanism dynamically adjusts the process noise covariance matrix based on the change amount of the fruit position between adjacent frames. When the system detects that the fruit position changes dramatically and the shaking amplitude is obvious, the process noise weight is automatically increased to increase the uncertainty range of the prediction to accommodate the mutation characteristics in the real motion. When the position of the fruit in the continuous frames is relatively stable or presents a periodic law, the noise covariance is reduced to enhance the prediction accuracy and stability. The adaptive adjustment mechanism based on displacement change driving makes the filter have environmental adaptability, which can dynamically adjust its prediction model under different fruit tree shapes, wind speed conditions and picking path disturbances, thereby maintaining efficient capture of the real motion state of the fruit. Motion trajectory prediction data is obtained.

[0068] In a specific embodiment, the process of performing step S105 can specifically include the following steps:

[0069] Based on the occlusion rate, the attitude angle in the growth attitude information, the depth value in the motion trajectory prediction data, the motion state and the complexity of the surrounding environment, a fruit picking factor set is created;

[0070] According to the fruit picking factor set, a picking suitability scoring function is constructed, and the fruit picking factor set is scored by using a scoring calculation model to obtain a picking suitability score;

[0071] Based on the picking suitability score, a picking decision is made to obtain a set of fruits to be picked;

[0072] For each fruit in the set of fruits to be picked, the position of the stem connection point is determined according to the segmentation result of the small field sensor, and the optimal picking point is calculated;

[0073] According to the optimal picking point and the picking suitability score, a picking execution instruction is generated.

[0074] Specifically, the fruit picking factor set is created based on the occlusion rate, the posture angle in the growth posture information, the depth value in the motion trajectory prediction data, the motion state and the surrounding environment complexity. The occlusion rate is derived from the fruit mask map generated by the small field segmentation network, which quantifies the target clarity by analyzing the area ratio of the fruit area and the occluded area. The posture angle is calculated by the included angle between the main direction of the contour and the vertical direction of the image, reflecting the inclination degree of the fruit and the complexity of the growth posture. The depth value is obtained by fusing the large field RGB-D image and the small field center pixel depth information, which is an important geometric parameter for evaluating the target distance. The motion state is analyzed based on the velocity and acceleration state vectors output by the Kalman filter prediction model, reflecting the stability and traceability of the fruit under external force. The environment complexity considers the leaf density, branch occlusion, light contrast and background interference features around the target, and is obtained by extracting the adjacent region feature map by the environment vision module and performing clustering analysis and complexity scoring modeling. The picking suitability scoring function is constructed according to the fruit picking factor set. The scoring function is obtained by setting the experience parameters of the picking experts and the regression modeling of the historical picking experimental data, in which each factor is assigned different weight coefficients in the function to reflect its influence on the actual picking success rate. For example, the occlusion rate and the posture angle have high weights, as they directly affect the feasibility of visual recognition and mechanical clamping. The depth value and the motion state are medium weight parameters, which are used to limit the accessibility and stability of the picking path. The environment complexity is an auxiliary weight, which is used for risk estimation and work time estimation. In the actual scoring process, the factor set of each fruit is input into the scoring function to calculate the picking suitability score value under a unified standard. This score value reflects whether the current target is in a pickable state in numerical value, and can form a prioritized system among multiple targets. When the picking suitability score results of all candidate target fruits are calculated, a set of threshold strategies is set for picking decision execution, marking the fruits with score values exceeding a certain threshold as the target to be picked, and these targets are unified into the set of fruits to be picked. This set is used as the input set for subsequent path planning, end clamping trajectory generation and picking action scheduling, and the target sorting will be arranged from high to low according to the score value to ensure that the fruits with high operation success rate, low loss and small path cost are prioritized. For each target in the set of fruits to be picked, the high-resolution mask map output by the small field segmentation network is called again to extract the connection region between the fruit and the stem through fruit contour point and edge structure analysis. This region is a long transition zone between the upper contour of the fruit and the background, which has obvious characteristics in edge gradient, color change and geometric shape.The position of the fruit stem connecting point is determined by edge intensity gradient analysis and shape prior discrimination algorithm, and a local coordinate system of the fruit is established near the point. In the coordinate system, the fruit stem direction is taken as the z-axis direction, the center position of the fruit and the local surface normal vector are combined to calculate the optimal picking point, which is the spatial contact position that can minimize the interference of clamping, ensure the picking angle and avoid obstacles. This position not only considers the minimization of the approaching path of the mechanical arm, but also integrates the geometric symmetry of the fruit and the stability of the fruit stem structure to ensure that the fruit does not fall off, break or twist during clamping. The optimal picking point and the picking suitability score corresponding to the target fruit are fused to generate a standardized picking execution instruction. The instruction includes the position coordinates and attitude direction parameters that the mechanical arm needs to reach, and also includes control variables such as grabbing speed, clamping force, action time window, etc. The scoring result determines whether to use a low-speed high-precision clamping mode (if the score is at the critical edge) or a high-speed fast operation mode (if the score is very high). For fruits with a score below the threshold but slightly higher than the discard boundary, the system determines whether their suitability can be improved after attitude adjustment. If so, it enters the waiting window queue, otherwise it is temporarily suspended for picking and re-evaluated in the next detection cycle.

[0075] The above describes the target fruit dynamic tracking method based on visual perception and deep learning in the embodiments of the present application. The following describes the target fruit dynamic tracking system based on visual perception and deep learning in the embodiments of the present application. Please refer to Figure 2 An embodiment of the target fruit dynamic tracking system based on visual perception and deep learning in the embodiments of the present application includes:

[0076] An image enhancement module 201 is configured to collect original image data and perform image enhancement through an adaptive regulation model of an encoder-decoder architecture to obtain an enhanced large-view image.

[0077] A target detection module 202 is configured to perform deep learning target detection on the enhanced large-view image, extract position information and picking preparation point positions of the fruit target, and guide a preset small-view sensor to the picking preparation point positions.

[0078] A pixel-level segmentation module 203 is configured to collect local image data and perform pixel-level segmentation through a multi-target segmentation network to obtain occlusion rate and growth posture information of the fruit target.

[0079] A dynamic tracking module 204 is configured to perform dynamic tracking on the fruit target according to the obtained depth information and segmentation results of the small-view sensor to obtain motion trajectory prediction data.

[0080] The computing module 205 is configured to calculate picking suitability scores and optimal picking points according to the occlusion rate, the growth posture information, and the motion trajectory prediction data, and generate picking execution instructions.

[0081] Through the cooperation of the above-mentioned components, through the cooperative operation of the large-view sensor and the small-view sensor, the technical bottleneck that a single visual sensor cannot simultaneously consider global perception and local accurate recognition is solved; multi-scale, multi-angle, and multi-resolution cooperative perception is achieved, and the fruit detection coverage rate and positioning accuracy in a complex orchard environment are significantly improved. The adaptive image regulation model using the encoder-decoder architecture effectively solves the influence of interference factors such as strong light, shadow, occlusion, and motion blur in the orchard environment on image quality; through two special processing branches, pixel-level enhancement and global ISP parameter learning are performed, so that the system can adapt to changing light conditions. The target detection and multi-target segmentation network based on deep learning enhances the feature extraction capability of the fruit; the multi-target instance segmentation network of the MASNet architecture realizes effective identification of semi-occluded fruits, and provides accurate occlusion rate and fruit posture information for subsequent picking decisions. The pseudo point cloud data and BEV view are constructed based on the RGB-D information, and the fruit positioning in the three-dimensional space is realized; the Kalman filtering algorithm with an adaptive process noise adjustment mechanism can accurately predict the motion trajectory of the fruit under wind disturbance, solving the problem of inaccurate tracking of dynamic targets in traditional methods. By comprehensively considering multiple key factors such as fruit occlusion rate, growth posture, depth value, motion state, and surrounding environment complexity, a scientific picking suitability score function is constructed; according to different picking difficulties, the working parameters of the end effector are automatically adjusted, overcoming the limitation of single picking point selection in the prior art.

[0082] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, system and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0083] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a target fruit dynamic tracking device based on visual perception and deep learning (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0084] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for dynamic tracking of target fruits based on visual perception and deep learning, characterized in that, include: Raw image data is acquired using a large field-of-view sensor in a cascaded vision system. The raw image data is then divided into raster regions to obtain a set of segmented image sub-regions. These sub-regions are input into an adaptive control model of an encoder-decoder architecture. Feature encoding is performed using a 5-layer convolutional structure in the encoder to obtain image encoded features. Feature decoding is then performed using a 5-layer transposed convolutional structure in the decoder to obtain initial decoding results. These initial decoding results are then input into the first processing branch for pixel-level enhancement to obtain pixel-level enhanced features. Finally, these initial decoding results are input into the second processing branch for ISP parameter learning to obtain global adjustment parameters. The pixel-level enhancement features are fused with global adjustment parameters to obtain an enhanced large field-of-view image; Deep learning is used to detect targets in the enhanced large field-of-view image, extracting the location information of the fruit target and the location of the picking preparation point, and guiding the pre-set small field-of-view sensor to the picking preparation point location. Local image data is collected and pixel-level segmented using a multi-target segmentation network to obtain the occlusion rate and growth posture information of the fruit target; RGB-D information is obtained based on depth information, and a depth value is assigned to each pixel. The two-dimensional image coordinates are converted into three-dimensional spatial coordinates using the camera intrinsic parameter matrix, resulting in pseudo-point cloud data for the fruit region. Based on this pseudo-point cloud data, a horizontal XOY coordinate system is established with the harvesting mechanism as the origin. The three-dimensional position of the fruit is projected onto the horizontal plane to form a BEV coordinate representation. The center point coordinates and corresponding depth values ​​of the fruit bounding box are extracted from the segmentation results of the small field-of-view sensor, and the three-dimensional spatial position of the fruit target is calculated using the BEV coordinate representation. Inter-frame matching is performed on the three-dimensional spatial position of the fruit target to obtain its motion trajectory. A Kalman filter algorithm is used to predict the state of the fruit target's motion trajectory, resulting in fruit state prediction data. Based on the fruit state prediction data, an adaptive process noise adjustment mechanism is introduced, and the process noise covariance matrix is ​​dynamically adjusted according to the change in fruit position in adjacent frames to obtain motion trajectory prediction data. A fruit picking factor set is created based on occlusion rate, posture angle from growth posture information, depth value from motion trajectory prediction data, motion state, and surrounding environmental complexity. A picking suitability scoring function is constructed based on this factor set, and a scoring calculation model is used to score the factor set to obtain a picking suitability score. Picking decisions are made based on the picking suitability score to obtain a set of fruits to be picked. For each fruit in the set, the location of the stem connection point is determined based on the segmentation results of the small field-of-view sensor. A local coordinate system is established on the fruit surface, and the optimal approach path and optimal picking point location are calculated based on the spatial relationship between this point location and the fruit's geometric center. A picking execution command is generated based on the optimal picking point and the picking suitability score.

2. The target fruit dynamic tracking method based on visual perception and deep learning according to claim 1, characterized in that, Before acquiring raw image data, the process includes: setting parameters for the large field-of-view sensor to ensure its field of view is greater than 120°, its resolution reaches 1920×1080 pixels, and its acquisition frequency is set to 30 frames / second; setting parameters for the small field-of-view sensor to control its field of view within the range of 30°-60°, its resolution reaches 2560×1440 pixels, and its acquisition frequency is set to 60 frames / second; rigidly connecting and installing the large and small field-of-view sensors, and determining the precise positional relationship between the sensors and the mechanical structure using a hand-eye calibration algorithm to obtain the installation positions of the large and small field-of-view sensors; based on the installation positions of the large and small field-of-view sensors, establishing the robot body base coordinate system, the robot picking mechanism coordinate system, the large field-of-view sensor coordinate system, the large field-of-view sensor target picking preparation point coordinate system, the small field-of-view sensor coordinate system, and the small field-of-view sensor target picking point coordinate system to obtain a cascaded vision system.

3. The target fruit dynamic tracking method based on visual perception and deep learning according to claim 2, characterized in that, Deep learning is used to detect targets in the enhanced large field-of-view image, extracting the location information of the fruit target and the position of the picking preparation point. A small field-of-view sensor is then guided to the picking preparation point. This process includes: inputting the enhanced large field-of-view image into a target detection network for depthwise separable convolution processing to obtain an initial feature map; performing channel and spatial attention calculations on the initial feature map using a CBAM attention module to obtain an enhanced feature map; performing multi-scale feature fusion detection on the enhanced feature map using three detection heads of different scales to obtain the location information of the fruit target; and applying non-maximum suppression to the location information. The system processes the data to obtain a precise bounding box for the fruit targets. Based on this precise bounding box, the coordinates of the center point of each fruit target are calculated and used as the coordinates of the pre-harvest point. Simultaneously, the depth value of the corresponding point is obtained from the large field-of-view sensor to obtain the pre-harvest point position. A coordinate transformation matrix is ​​then applied to the pre-harvest point position, sequentially transforming it from the large field-of-view sensor coordinate system to the robot body base coordinate system, and then to the robot harvesting mechanism coordinate system, to obtain the position parameters for controlling the movement of the small field-of-view sensor. Based on these position parameters, the small field-of-view sensor is guided to the pre-harvest point position.

4. The target fruit dynamic tracking method based on visual perception and deep learning according to claim 3, characterized in that, Local image data is acquired and pixel-level segmented using a multi-object segmentation network to obtain the occlusion rate and growth posture information of the fruit target. This includes: acquiring local image data through a small-field-of-view sensor in a cascaded vision system, inputting the local image data into the multi-object segmentation network for pixel-level mask segmentation to obtain an initial pixel-level mask; using a ResNext-101-FPN structure, multi-scale feature map extraction is performed on the initial pixel-level mask to obtain a multi-scale feature representation; bounding box regression and classification calculations are performed on the multi-scale feature representation to obtain the precise bounding box of the fruit target; and pixel-level mask generation is performed on the multi-scale feature representation based on the precise bounding box of the fruit target to obtain a precise segmentation mask of the fruit target; the occlusion rate of the fruit target is calculated based on the precise segmentation mask, and the angle between the fruit's posture angle and the vertical direction is calculated based on the precise segmentation mask to obtain the growth posture information of the fruit target.

5. A dynamic tracking system for target fruits based on visual perception and deep learning, characterized in that, The method for dynamically tracking target fruits based on visual perception and deep learning as described in any one of claims 1-4 includes: The image enhancement module is used to acquire raw image data and perform image enhancement through an adaptive control model of encoder-decoder architecture to obtain an enhanced large field-of-view image; The target detection module is used to perform deep learning target detection on the enhanced large field-of-view image, extract the location information of the fruit target and the position of the picking preparation point, and guide the preset small field-of-view sensor to the position of the picking preparation point. The pixel-level segmentation module is used to acquire local image data and perform pixel-level segmentation through a multi-target segmentation network to obtain the occlusion rate and growth posture information of the fruit target. The dynamic tracking module is used to dynamically track the fruit target based on the acquired depth information and the segmentation results of the small field-of-view sensor, and obtain motion trajectory prediction data. The calculation module is used to calculate the harvesting suitability score and the optimal harvesting point based on the shading rate, growth posture information and movement trajectory prediction data, and generate harvesting execution instructions.

Images