Cold region greenhouse multispectral visual positioning picking method and system

Abstract

The invention provides a cold region greenhouse multispectral visual positioning picking method and system. The method comprises the following steps: synchronously acquiring an RGB image, an NIR image, greenhouse environment temperature and humidity data and depth information of a target; based on the RGB image, the NIR image and the environment data, image correction is carried out through an image correction network, and a sharpened RGB image is output; inputting the corrected RGB image, NIR image and depth image into a maturity identification model for processing, and outputting the maturity grade and three-dimensional space coordinates of the fruit; based on the three-dimensional coordinates and maturity information of the fruits and a pre-constructed environment three-dimensional point cloud map, a collision-free optimal picking path is planned for a mechanical arm; the mechanical arm is controlled to execute the picking action, the success rate of the picking result and the recognition confidence coefficient serve as feedback signals, and online fine adjustment is conducted on parameters of the image correction network and the maturity recognition model; the problems of low picking efficiency, low precision and high damage rate of the greenhouse in the cold region are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picking positioning, in particular to a multi-spectral vision positioning picking method and system for cold greenhouse. BACKGROUND

[0002] In the process of agricultural modernization, facility agriculture as an efficient production mode plays a key role in cold regions (such as Northeast China, high-latitude cold regions). The cold climate has the characteristics of low temperature in winter, weak light intensity, large diurnal temperature difference, and easy formation of high humidity environment inside the greenhouse. These environmental conditions pose severe challenges to the picking operation of greenhouse crops (especially fruit and vegetable crops). The traditional picking mode and existing technology gradually expose the insurmountable bottlenecks, which are embodied in the following aspects:

[0003] The internal environment of the cold greenhouse is significantly different from that of the normal temperature greenhouse, and its special conditions directly affect the efficiency and accuracy of the picking operation:

[0004] In winter, the natural light in the cold region is insufficient, and the greenhouse relies on artificial light, which is prone to uneven illumination. At the same time, the high humidity environment makes the air contain a lot of water vapor, which causes the RGB images collected by the imaging device to have a hazy scattering effect, the image clarity decreases, and the color distortion occurs. For example, the red saturation of tomato fruits decreases, and the contrast with green leaves weakens, making it difficult to accurately distinguish the fruit outline and maturity characteristics. In addition, the high humidity environment also affects the stability of the optical sensor, causing the collected image noise to increase, further interfering with the subsequent feature recognition.

[0005] At night or in low-temperature periods in the cold greenhouse, the indoor temperature is often lower than 10℃, and the driving components (such as motors and hydraulic devices) of some flexible mechanical arms are prone to problems such as decreased deformation accuracy and delayed motion response in low-temperature environments. At the same time, the brittleness of crop leaves and fruits increases in low-temperature environments, which can easily cause fruit damage or leaf breakage, reducing the picking quality.

[0006] In order to ensure crop growth, the cold greenhouse often uses multi-layer planting or three-dimensional cultivation mode, and the branches and leaves of crops are densely intertwined, and the fruits are often shaded by leaves or in complex spatial positions. In addition, the leaves of crops in cold regions increase in thickness and the fruit skin increases in toughness to adapt to the low-temperature environment. The traditional maturity judgment method based on color (such as simply relying on the color threshold of RGB images) is prone to misjudgment and cannot accurately identify the internal quality of fruits (such as sugar content and maturity uniformity). SUMMARY

[0007] Based on the above problems, the present application provides a cold region greenhouse multispectral visual positioning picking method and system, which can adapt to the dynamic environment of the cold region, fuse multi-modal information, realize accurate image correction through synchronous acquisition of multispectral images and environmental data, improve maturity recognition accuracy through a cross-modal attention fusion network, improve picking success rate through a path planning algorithm considering the deformation constraint of a flexible mechanical arm, and optimize model parameters through a closed-loop feedback mechanism, thereby solving the problems of low picking efficiency, low precision and high damage rate in the cold region greenhouse.

[0008] In a first aspect, the present application provides a cold region greenhouse multispectral visual positioning picking method, which comprises:

[0009] S1: synchronously acquiring an RGB image, an NIR image, greenhouse environmental temperature and humidity data, and depth information of a target;

[0010] S2: based on the RGB image, the NIR image, and the environmental data, performing image correction through an image correction network to output a clear RGB image; the image correction network dynamically modulates the feature fusion weight of the RGB and the NIR through an environmental prior vector, and its training process is constrained by an atmospheric scattering physical model;

[0011] S3: inputting the corrected RGB image, the NIR image, and the depth image into a maturity recognition model for processing, the maturity recognition model extracts and fuses color, texture, and internal quality features through a cross-modal attention fusion network, and outputs the maturity grade and three-dimensional spatial coordinates of the fruit;

[0012] S4: based on the three-dimensional coordinates, the maturity information of the fruit, and a pre-constructed environmental three-dimensional point cloud map, an optimal picking path without collision is planned for a mechanical arm;

[0013] S5: controlling the mechanical arm to perform picking actions, and taking the success rate and recognition confidence of the picking result as feedback signals to perform online fine-tuning on the parameters of the image correction network and the maturity recognition model.

[0014] Preferably, S2 comprises:

[0015] S21: reading environmental temperature and humidity sensor data and calculating a definition evaluation index of the RGB image to form an environmental prior vector together;

[0016] S22: inputting the environmental prior vector into a parameter prediction network to generate a dynamic weight matrix corresponding to the RGB features and the NIR features; using the weight matrix to perform weighted fusion on the extracted RGB color texture features and NIR structure edge features; and generating a preliminary fusion feature map;

[0017] S23: inputting the preliminary fusion feature map into a generator network to generate a corrected image.

[0018] Preferably, the generator is optimized by a fusion loss function, which includes an adversarial loss, a perception loss, and a physical consistency loss; the physical consistency loss term is constructed based on an atmospheric scattering model, and is used to constrain the relationship among the generated image, the input image, the estimated transmittance map, and the atmospheric light.

[0019] Preferably, the clarity evaluation index is an image average gradient and / or an image entropy.

[0020] The parameter prediction network is a multi-layer perception machine.

[0021] The dynamic weight matrix is a channel-level weight vector or a spatial-level attention map.

[0022] Preferably, the maturity recognition model is a multi-branch network architecture, comprising:

[0023] A first encoding branch for extracting color and texture features from the corrected RGB image;

[0024] A second encoding branch for extracting internal quality features from the NIR image;

[0025] A cross-modal attention fusion module for interacting and fusing the features extracted by the multiple branches; the cross-modal attention fusion module is configured to enable the features of any modality to serve as a query vector to interact with the features of another modality to retrieve and focus on complementary information therein;

[0026] A classifier for outputting a maturity level based on the fused features.

[0027] Preferably, the maturity recognition model further comprises a third encoding branch for extracting spatial distance and geometric shape features from the depth image.

[0028] The cross-modal attention fusion module is configured to use the distance features extracted by the third encoding branch as a shared query vector to guide and modulate the attention degree of the first encoding branch to the color and texture features of the fruit edge, and the attention degree of the second encoding branch to the internal quality features of the fruit center.

[0029] The execution process of the cross-modal attention fusion module includes generating a spatial weight mask according to the depth features, and performing mask weighting on the RGB features to strengthen the response of the fruit edge area, and performing reverse mask weighting on the NIR features to strengthen the response of the fruit center area.

[0030] Preferably, before the image is input into the maturity recognition model, a preprocessing step is further included.

[0031] contrast-limited adaptive histogram equalization is performed on the corrected RGB image and the NIR image, and the RGB image and the NIR image are respectively pixel-level registered with the depth image.

[0032] Preferably, the mechanical arm is a flexible mechanical arm, and the S4 comprises:

[0033] When planning a path for the flexible mechanical arm, the physical deformation constraint of the flexible mechanical arm is taken as a constraint condition for path searching; the physical deformation constraint includes avoiding deformation curvature and avoiding bending deformation angle;

[0034] In the path set satisfying the constraint condition, the optimal path from the starting point of the mechanical arm to the target point of the fruit is selected.

[0035] Preferably, the path planning adopts an algorithm based on spatial segmentation, comprising:

[0036] Taking a set of obstacle space points in a pre-constructed three-dimensional point cloud map of the environment as a root node, the workspace of the mechanical arm is recursively body segmented until the volume of a sub-region reaches a preset minimum value, forming a discrete, passable and flexible deformation constraint satisfying path network, and path searching is performed in the network.

[0037] In the second aspect, the present application provides a cold greenhouse multispectral visual positioning and picking system for the cold greenhouse multispectral visual positioning and picking method described in the embodiments of the present application, and the system comprises:

[0038] A multispectral visual perception module comprising at least one RGB-NIR multispectral camera, at least one depth camera, and a temperature and humidity sensor, is used to synchronously collect RGB images, NIR images, depth information and greenhouse environment temperature and humidity data of a target;

[0039] A control module with a built-in processor and memory is used to store and run the image correction network, maturity recognition model and path planning algorithm, and generate control instructions;

[0040] An execution module comprising a multi-degree-of-freedom mechanical arm and a picking executor at the end thereof is used to receive the control instructions and complete picking actions.

[0041] Compared with the prior art, the beneficial effects of the present application at least include: by synchronously collecting the greenhouse environment temperature and humidity data and the RGB, NIR images, constructing the environment prior vector dynamic modulation feature fusion weight, and combining the atmospheric scattering physical model to constrain the image correction network training, the correction parameters can be adjusted in real time according to the humidity fluctuation of the cold region greenhouse, the image fog in the high humidity environment is effectively removed, the clarity of the corrected RGB image is improved, the color distortion rate is reduced, and the subsequent feature extraction is not affected by the low temperature and high humidity environment; the image correction network is constrained by the environment prior vector and the physical model, and the overfitting problem of the existing data-driven model is avoided.

[0042] The maturity recognition model extracts RGB color texture, NIR internal quality and depth spatial features through multiple coding branches respectively, and uses a cross-modal attention module to enable precise interaction between different modal features, effectively solving the problem that existing single modal models cannot consider internal and external quality and misjudgment of overlapping fruits, improving the accuracy of maturity recognition, and reducing the missing rate and over-sampling rate.

[0043] The corrected RGB and NIR images are enhanced by adaptive histogram equalization (CLAHE), and pixel-level registration is combined with the depth image to solve the problem of insufficient contrast caused by low light in the cold region, improve the distinction between fruits and leaves, and avoid feature loss caused by branch and leaf shielding or uneven light, providing high-quality data support for maturity recognition and three-dimensional positioning.

[0044] The physical deformation constraints (avoidance deformation curvature and bending deformation angle) of the flexible manipulator are introduced in the path planning, and the space is segmented based on the environment three-dimensional point cloud map to construct a passable network, which reduces the collision rate of the manipulator and the branches and leaves compared with the existing fixed path planning; at the same time, the flexible manipulator is adapted to the increased brittleness of crops (fruits, leaves) at low temperature in the cold region, reducing the damage rate of fruit extrusion, and is especially suitable for picking soft fruits and vegetables such as strawberries and tomatoes.

[0045] The working space of the manipulator is quickly dispersed through recursive body segmentation to form a passable path network, shorten the path planning time, meet the real-time requirements of large-scale picking in the cold region greenhouse (the picking cycle of a single plant is controlled within 10 seconds), and improve the picking efficiency.

[0046] The picking success rate and recognition confidence are used as feedback signals to fine-tune the image correction network and maturity recognition model online, which can dynamically adapt to long-term changes in the cold greenhouse environment (such as changes in light intensity caused by seasonal alternation and changes in morphology during the growth cycle of crops), reduce the model accuracy decay rate, and reduce the need for frequent manual model calibration, thereby reducing the later operation and maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of the multispectral visual positioning and picking method of the cold greenhouse embodiment of the present application. DETAILED DESCRIPTION

[0048] In view of the deficiencies in the prior art, the present applicant has obtained the technical solution of the present application through long-term research and a large number of practices. The technical solution, its implementation process and principles will be further explained in the following in combination with the drawings and specific implementation cases in the embodiments of the present application.

[0049] It should be noted that the embodiments described below by referring to the drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation on the present application. The described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, any alternative, modification, equivalent method and solution within the spirit, principles and scope of the present application defined by the claims, and all other embodiments obtained by those skilled in the art without creative labor, all fall within the scope of protection of the present application.

[0050] Further, in order for the public to have a better understanding of the present application, some specific details are described in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0051] Embodiment 1: refer to the drawings Figure 1 The embodiments of the present application provide a cold region greenhouse multi-spectral visual positioning picking method, comprising:

[0052] S1: synchronously collecting the RGB (visible light) image, NIR (near infrared) image, environmental temperature and humidity data and depth information of the target in the greenhouse environment; wherein the greenhouse environment can be a low-temperature and high-humidity environment;

[0053] S2: based on the RGB image, NIR image and environmental data, image correction is performed through an image correction network to output a clear RGB image; the image correction network dynamically modulates the feature fusion weight of RGB and NIR through an environmental prior vector, and its training process is constrained by an atmospheric scattering physical model;

[0054] S3: inputting the corrected RGB image, NIR image and depth image into a maturity recognition model for processing, the maturity recognition model extracts and fuses color, texture and internal quality features through a cross-modal attention fusion network, and outputs the maturity grade and three-dimensional spatial coordinates of the fruit;

[0055] S4: based on the three-dimensional coordinates, maturity information of the fruit and the pre-constructed environmental three-dimensional point cloud map, an optimal picking path without collision is planned for a flexible mechanical arm;

[0056] S5: controlling the flexible mechanical arm to perform a picking action, and taking the success rate of picking and the recognition confidence as feedback signals to fine-tune the parameters of the image correction network and the ripeness recognition model online.

[0057] The working principle of the above technical solution is:

[0058] The RGB camera is used to collect the appearance color and texture information of the fruit, capture the visual features related to the surface ripeness of the fruit, such as the proportion of red area when the tomato is ripe; the near-infrared (NIR) camera is used to collect the internal quality information of the fruit, reflecting the internal ripeness indicators such as sugar content and moisture content, for example, the absorption intensity of specific wavelengths in the near-infrared spectrum is positively correlated with sugar content; the temperature and humidity sensor is used to obtain the greenhouse environment data in real time, providing environmental adaptation basis for subsequent image correction; the depth camera (such as TOF camera) is used to collect the three-dimensional spatial distance information of the fruit, determining the position coordinates of the fruit relative to the mechanical arm. The above data is synchronized through time stamp synchronization technology to ensure that each group of data corresponds to the same picking target and environmental state, providing complete and matched basic data for subsequent processing.

[0059] The collected environmental temperature and humidity data and the RGB image clarity evaluation index are fused to construct an environmental prior vector, which can quantify the influence of the current environment on image quality, for example, high humidity corresponds to high scattering interference, and low light corresponds to low clarity; then the environmental prior vector is input into the parameter prediction network to dynamically generate a fusion weight matrix of RGB features and NIR features, for example, in high humidity environment, the weight of NIR features (strong anti-scattering ability) is increased, and in low light environment, the weight of the brightness channel in RGB features is increased, and the advantages of the two modalities are complementary through weighted fusion to generate a preliminary fusion feature map; at the same time, in the training process of the image correction network, the atmospheric scattering physical model is introduced as a constraint to ensure that the corrected RGB image meets the real optical propagation law, for example, the relationship between the pixel brightness of the corrected image and the atmospheric light and transmittance satisfies the scattering model formula, avoiding the distortion caused by data-driven correction, and finally outputting a clear RGB image that can accurately reflect the appearance features of the fruit.

[0060] To break through the limitations of single modal that cannot consider both fruit appearance and internal quality and cannot determine the spatial position, a cross-modal attention fusion recognition logic is adopted, and the corrected RGB image, original NIR image and depth image are input into the maturity recognition model. The model first processes the three types of data through different feature extraction branches. The appearance features such as color (e.g. R / G ratio) and texture (e.g. gray level co-occurrence matrix features) are extracted from the RGB image, the internal quality features such as spectral reflectance of specific wavelengths are extracted from the NIR image, and the three-dimensional spatial coordinate features are extracted from the depth image. Then, the feature interaction is realized through the cross-modal attention fusion network. Finally, the fused features are input into the classifier to output the fruit maturity grade (e.g. unripe, ripe, overripe), and the three-dimensional spatial coordinates of the fruit are calculated based on the depth camera intrinsic parameters to realize the integration of maturity judgment and spatial positioning.

[0061] Based on the pre-constructed three-dimensional point cloud map of the greenhouse environment, the spatial distribution of obstacles (such as branches, leaves and supports) is determined. Then, combined with the three-dimensional coordinates and maturity information of the fruits output by the S3 (preferably picking fruits with high maturity), the physical characteristics of the flexible manipulator (such as deformation limit) are used as constraint conditions (such as bending angle not exceeding the safety threshold and deformation curvature meeting the material characteristics), and the feasible path from the starting position of the manipulator to the target position of the fruit is searched in the three-dimensional point cloud map. The optimal path is selected from the feasible path through the path evaluation function (such as shortest path length and minimum manipulator deformation), ensuring that the path avoids all obstacles and meets the motion constraints of the flexible manipulator, avoiding fruit damage and mechanical failure caused by collision or excessive deformation.

[0062] The picking results (such as whether the fruit is successfully grabbed and whether the fruit is damaged) are recorded and the success rate is calculated, and the recognition confidence (such as the predicted probability of maturity grade) output by the maturity recognition model is extracted. The success rate and confidence are used as feedback signals. If the success rate is lower than the preset threshold (e.g. 85%) or the confidence is lower than the threshold (e.g. 0.7), the parameters of the image correction network and the maturity recognition model are fine-tuned online, for example, if the corrected image quality decreases due to increased humidity, the success rate of picking decreases, the weight coefficient of the environmental prior vector in the parameter prediction network is adjusted; if the maturity recognition deviates due to the change of fruit variety, the interaction coefficient of the cross-modal attention fusion module is fine-tuned. Through continuous iteration and optimization, the model is always adapted to the dynamic environment of the cold greenhouse, and the high picking accuracy is maintained.

[0063] The effects of the above technical solutions are:

[0064] Through multi-source data synchronous acquisition and environment adaptive correction, the image quality problem caused by low temperature, high humidity and low light in cold regions is solved; the path planning fully considers the characteristics of the flexible mechanical arm and the environmental obstacles, and reduces the mechanical arm collision rate through the three-dimensional point cloud map and deformation constraint; through the real-time feedback of the picking success rate and the recognition confidence, the model parameters can be dynamically adjusted according to the environmental changes (such as seasonal alternation and crop growth), and the model precision decay rate is reduced.

[0065] In a possible implementation, the S2 includes:

[0066] S21, reading the environment temperature and humidity sensor data, and calculating the definition evaluation index of the RGB image, which together constitute an environment prior vector;

[0067] S22: inputting the environment prior vector into a parameter prediction network to generate a dynamic weight matrix corresponding to the RGB feature and the NIR feature; using the weight matrix to perform weighted fusion on the extracted RGB color texture feature and NIR structure edge feature; and generating a preliminary fusion feature map;

[0068] S23: inputting the preliminary fusion feature map into a generator network to generate a corrected image.

[0069] In a possible implementation, the generator is optimized through a fusion loss function, and the loss function includes: an adversarial loss, a perception loss, and a physical consistency loss; the physical consistency loss term is constructed based on an atmospheric scattering model, and is used to constrain the relationship among the generated image, the input image, the estimated transmittance map and the atmospheric light.

[0070] The definition evaluation index is an image average gradient and / or an image entropy;

[0071] The parameter prediction network is a multi-layer perception (MLP);

[0072] The dynamic weight matrix is a channel-level weight vector or a spatial-level attention map.

[0073] In application, the environment prior vector converts the unquantifiable heavy fog and high humidity concept into a specific digital vector that can be understood and processed by the neural network, and the specific construction steps are as follows:

[0074] Directly reading the numerical value of the sensor. Usually at least two scalars are included: the environment temperature temp (unit: ℃), and the environment relative humidity rh (unit: %);

[0075] The RGB image definition evaluation index is obtained by calculating the currently collected RGB image; common indexes include:

[0076] Average gradient grad: Reflects image details and sharpness, fog will cause the gradient value to decrease.

[0077] Image entropy entropy: Reflects the richness of image information; fog will cause the entropy value to decrease.

[0078] Dark channel prior intensity dark: The dark channel value of foggy images is higher.

[0079] The above scalar data is spliced into a one-dimensional vector. For example, a most basic prior vector P can be:

[0080] P = [temp, rh, grad]

[0081] A more complex prior vector P can be:

[0082] P = [temp, rh, grad, entropy, dark]

[0083] Before inputting the parameter prediction network, each dimension of the vector P must be normalized (e.g., scaled to a mean of 0 and a variance of 1, or scaled to the [0, 1] interval) to ensure training stability and convergence speed.

[0084] The parameter prediction network dynamically generates weights for controlling RGB and NIR feature fusion according to the environmental prior vector P; the parameter prediction network can be a small, lightweight Multilayer Perceptron (MLP). Its core role is to serve as a spectral fusion strategy controller.

[0085] Parameter prediction network structure example:

[0086] Input layer: number of neurons = dimension of environmental prior vector P (e.g., 3 or 5).

[0087] Hidden layer: usually 1 to 2 layers, with fewer neurons per layer (e.g., 64 or 128), using activation functions such as ReLU.

[0088] Output layer: generates weights for controlling RGB and NIR feature fusion.

[0089] The first implementation of the output layer is channel-level weights (global adaptive type):

[0090] The RGB image and the NIR image each contain feature channels (which can be understood as layers in the image that carry different information, such as some channels responsible for presenting red information and some responsible for texture details). A global uniform trust score is assigned to each of the feature channels. The network determines which channel information is more reliable according to the current greenhouse environment (such as humidity and fog intensity), and gives a higher trust score to these channels. The specific steps include:

[0091] First, determine how many feature channels the RGB image and the NIR image each have (for example, the RGB image may have 32 channels, and the NIR image may have 16 channels). The total number of trust scores to be generated for these two types of channels is the sum of the number of channels in each type (32+16=48).

[0092] A parameter prediction network (similar to a small intelligent judgment module) generates 48 trust scores according to the current environmental data (temperature and humidity, image clarity). These scores will be controlled between 0 and 1, with 0 representing complete distrust of the channel information and 1 representing complete trust.

[0093] The first 32 scores are assigned to the RGB channels, and the last 16 scores are assigned to the NIR channels. Then multiply the information of each channel by the corresponding trust score, and finally add the processed information of RGB and NIR to obtain the fused image information.

[0094] For example, in a heavy fog environment, the channel responsible for blue information in the RGB image will become very unreliable due to the scattering of fog, and the network will give this channel a trust score close to 0. The channels of the NIR image are less affected by fog, and the information is more reliable, so the network sets a high trust score for all NIR channels, making NIR information the main source of fused information.

[0095] The second implementation of the output layer is spatial-level weight (local fine-grained):

[0096] Compared to the global uniform distribution of channel-level weight, spatial-level weight is more fine-grained - it will judge for each pixel point in the image (which can be understood as each small grid in the image) whether to believe the RGB pixel information or the NIR pixel information, and assign a local trust score to each pixel point. It can accurately respond to different areas in the image that are affected by different degrees of environment (such as heavy fog in the distance and light fog in the near distance): the specific steps include:

[0097] If a weight is generated for each pixel of the image directly, the number of parameters to be processed will be very large (for example, an image of 256x256 pixels requires 65536 parameters), which will increase the network burden. Therefore, a parameter prediction network is used to generate a concise feature vector (similar to a string of condensed environmental judgment signals, such as containing 64 key information points) from the environmental data.

[0098] The condensed feature vector is input into a lightweight attention generation network (similar to a fine allocation module), which amplifies the string of signals into a weight map that is exactly the same size as the original image through technical means. Each pixel on the map has a value between 0 and 1, which is the local trustworthiness of the pixel.

[0099] For each pixel in the image, multiply the local trustworthiness on the weight map by the RGB image information at that pixel, multiply the local trustworthiness by the NIR image information at that pixel, and finally add the two results to get the fused information of the pixel. After all the pixels are processed, the final fused image is formed.

[0100] For example, the crop area in the distance of the image is far away and has a heavy fog, so the RGB pixel information is blurred and unclear. The network will set a low local trustworthiness (such as 0.2) for these distant pixels, meaning that only 20% of the RGB information is trusted, while 80% of the NIR information is trusted. The edge area of the fruit in the image is close, and the RGB can clearly present the details, with little influence from the fog. The network will set a high local trustworthiness (such as 0.8) for these close pixels, mainly relying on RGB information to preserve edge details, so that the fused image can clearly present the details of the nearby area and accurately restore the information of the distant area.

[0101] Through multi-dimensional loss function constraints, it is ensured that the generated correction image is real, clear and consistent with the physical law. The preliminary fusion feature map obtained in S22 is input into the generator network, and the core function of the generator is to convert the fusion feature into a visually clear RGB correction image. To avoid distortion of the generator output, the fusion loss function is used to optimize it, which includes three key parts:

[0102] Adversarial loss: the image generated by the generator is as similar as possible to the real clear image, avoiding obvious artificial synthesis traces (such as color discontinuity and blurred details).

[0103] Perception loss: constrain the image from the perspective of human visual perception, and the generated correction image is consistent with human perception of clear images in terms of details and textures (if the edge is sharp and the leaf texture is distinguishable).

[0104] Physical consistency loss: Based on the atmospheric scattering model (which can reflect the influence of water vapor, fog and other environmental factors on light propagation), it is used to constrain the relationship between the generated image, the original input image, and the estimated transmittance map (reflecting the ability of light to penetrate the fog) and the atmospheric light (the basic light intensity in the environment), ensuring that the corrected image is not generated in vain, but conforms to the real optical and physical laws (such as in a high humidity environment, the brightness and contrast changes of the corrected image are consistent with the predicted results of the atmospheric scattering model). Finally, under the constraint of the multi-dimensional loss function, the generator outputs a high-quality, distortion-free corrected image.

[0105] The above technical solutions have the following effects:

[0106] Through the environmental prior vector, the influence of temperature and humidity fluctuations and light changes in the greenhouse on the image can be captured in real time (such as the difference between high humidity in the morning and low humidity in the afternoon); combined with the dynamic weight matrix, the fusion of RGB and NIR features can be flexibly adjusted according to environmental changes, effectively solving the image blurring problem caused by fog and low light.

[0107] The fusion of the adversarial loss and the perceptual loss in the loss function ensures that the corrected image is visually clear and natural, consistent with human eye perception; the physical consistency loss ensures that the correction process follows the atmospheric scattering physical law, avoiding over-correction or under-correction (fog not completely removed) problems, providing a high-quality data basis for subsequent maturity recognition and fruit positioning.

[0108] The channel-level weight calculation speed is fast, suitable for large-scale greenhouse picking scenarios with high real-time requirements (such as processing more than 50 frames of images per minute); the spatial-level attention map can achieve pixel-level fine correction, suitable for scenarios with dense fruit and high edge detail requirements (such as strawberry, cherry and other small fruit and vegetable picking). The two methods can be flexibly selected according to the type of crops in the greenhouse and the efficiency requirements of picking, improving the adaptability of the technology.

[0109] The parameter prediction network uses a multi-layer perceptron (MLP), which has a simple structure and fewer parameters compared to complex deep learning networks. It can run efficiently on embedded devices commonly used in cold greenhouse environments (such as edge computing modules), without the need for high-performance servers, reducing the cost of device deployment. At the same time, the training and iteration speed of MLP is fast, which can quickly adapt to the environmental characteristics of different cold greenhouses (such as the difference in temperature and humidity between Northeast and Northwest China), shortening the model debugging cycle.

[0110] In one possible implementation, the maturity recognition model is a multi-branch network architecture, including:

[0111] The first encoding branch is configured to extract color and texture features from the corrected RGB image. The features are extracted from the RGB image layer by layer through a network structure including a convolution layer and a pooling layer. The shallow network captures basic color information (for example, the hue change of green, pink, and red during the ripening process of tomatoes, and the proportion of the red distribution area on the surface of strawberries). The deep network extracts more complex texture features (for example, the smoothness of the strawberry skin, the subtle texture changes when the ripeness is high, or the rough texture of the unripe fruit). The color and texture features that are valuable for maturity judgment are screened through an activation function (for example, ReLU) and normalization processing, and noise interference (for example, local color deviation caused by leaf shadows and texture deformation caused by shooting angles) is suppressed. Finally, a color and texture feature vector that can clearly reflect the appearance state of the fruit is output.

[0112] The second encoding branch is configured to extract internal quality features from the NIR image. The NIR image can reflect the absorption and reflection differences of internal substances (such as sugar, water, and starch) of the fruit to near-infrared light. For example, during the ripening process of the fruit, starch is gradually converted into sugar, and the reflectivity of a specific wavelength (for example, 900-1100 nm) in the NIR spectrum will significantly increase. The unripe fruit has a high water content, and the absorption rate of a specific wavelength is higher. The second encoding branch captures these spectrum features that are strongly related to internal quality through a special convolution kernel and a feature extraction module. The extracted near-infrared features are converted into a quantifiable feature vector, and a corresponding relationship between the feature value and the internal quality is established (for example, the higher the reflectivity value, the higher the sugar content and the higher the ripeness). Finally, an internal quality feature vector that can represent the internal ripening state of the fruit is output.

[0113] The cross-modal attention fusion module is configured to interact and fuse the features extracted by the multiple branches. The cross-modal attention fusion module is configured to enable the features of any modality to serve as a query vector to interact with the features of another modality to retrieve and focus on complementary information.

[0114] The module first takes the color and texture feature vector output by the first branch as a query vector to retrieve the internal quality feature vector of the second branch. For example, when the RGB feature shows that the fruit appearance is dark red (appearance mature), the NIR feature is queried to determine whether the sugar content meets the standard. If the internal sugar content is insufficient (NIR feature shows unripe), the fruit is judged as appearance mature and internal unripe, avoiding misjudgment based on appearance alone. At the same time, the module also takes the internal quality feature vector of the second branch as a query vector to retrieve the color and texture feature vector of the first branch. For example, the NIR feature shows that the internal sugar content of the fruit has reached the standard (internal mature), but the RGB feature shows that the appearance is still green (appearance unripe). Further verification is needed to determine whether the RGB feature is affected by environmental interference (such as color deviation caused by leaf shading). If the interference is excluded and the appearance is indeed unripe, it is judged as internal early maturity and appearance unripe, and the harvest should be delayed.

[0115] Through attention mechanism, the features more relevant to maturity judgment in the two modalities are given higher weights (for example, when the appearance is unobstructed, the RGB color feature weight is high; when the internal quality difference is significant, the NIR feature weight is high). The weighted feature vectors are then fused into appearance and internal collaborative feature vectors to ensure that the fused features contain both appearance details and internal quality, providing more comprehensive information.

[0116] A classifier is used to output the maturity level based on the fused features. The classifier has a feature and maturity mapping model trained with a large number of samples, and also outputs the prediction probability of each maturity level.

[0117] The above technical solution has the following effects:

[0118] Traditional models rely solely on RGB appearance features for recognition, which can lead to misjudgments such as "appearance mature and internal unripe" and "appearance unripe and internal mature". This model extracts appearance and internal features through dual branches and then performs cross-modal fusion to achieve "internal and external consideration". In tests of cold greenhouse tomatoes, strawberries, and other crops, the maturity recognition accuracy can reach more than 95%, which is 20-25 percentage points higher than single RGB models. The miss rate (judging mature fruit as unripe) and the false rate (judging unripe fruit as mature) are both controlled within 3%.

[0119] The cross-modal attention query mechanism can effectively deal with these disturbances in the complex cold greenhouse environment (such as low light causing RGB color distortion and leaf shading causing local NIR feature loss).

[0120] The first and second coding branches can be flexibly adjusted according to the characteristics of different crops (for example, for strawberries, the RGB branch strengthens the extraction of the distribution features of red spots on the surface, and the NIR branch optimizes the feature capture of sugar-sensitive wavelengths; for cucumbers, the RGB branch strengthens the features of the green depth of the skin, and the NIR branch optimizes the features of water content), without having to reconstruct the entire model.

[0121] In one possible implementation, the maturity recognition model further includes a third encoding branch for extracting spatial distance and geometric features from the depth image; through deep convolutional layers and pooling layers, it extracts the relative distance features between the fruit and the robotic arm from the depth image. For example, it converts the distance value (in millimeters, for example) of each pixel in the depth image into a feature vector, clarifies the spatial coordinates of the fruit (e.g., 500mm from the camera, 100mm horizontal offset), and distinguishes the distance differences between different fruits (e.g., the distance value of nearby fruits is small, and the distance value of distant fruits is large), avoiding the positioning deviation of mistaking distant fruits for nearby targets due to fruit overlap.

[0122] By using edge detection modules (such as the Sobel operator and contour extraction algorithm) and shape descriptors (such as roundness and aspect ratio), the geometric structural features of the fruit are extracted from the depth image; at the same time, the occlusion state of the fruit is identified, and a distance-geometric feature vector that can characterize the spatial morphology of the fruit is output.

[0123] The cross-modal attention fusion module is configured to: use the distance features extracted by the third coding branch as a shared query vector to guide and modulate the attention of the first coding branch to the color and texture features of the fruit edge, and the attention of the second coding branch to the quality features inside the center of the fruit.

[0124] The execution process of the cross-modal attention fusion module includes: generating a spatial weight mask based on depth features, applying mask weighting to RGB features to enhance the response of the fruit edge region, and applying reverse mask weighting to NIR features to enhance the response of the fruit center region.

[0125] Based on the existing two-way interaction logic, a new deep feature guidance mechanism is added, making spatial information a bridge connecting appearance (RGB) and internal (NIR) features, solving the problems of feature confusion in overlapping fruits and inaccurate regional feature weight allocation. The specific execution process is as follows:

[0126] First, the distance-geometric feature vector output from the third branch is transformed into a spatial weight mask. This is a region weight map with the exact same size as the RGB and NIR images. The weight value (between 0 and 1) of each pixel corresponds to the spatial priority of that region.

[0127] Fruit edge region: Based on the geometric contour of the depth image (such as abrupt changes in the depth value of edge pixels), the weight values ​​of the fruit edge pixels are set to high (such as 0.8-1.0), and the weight values ​​of non-edge regions (such as leaves and background) are set to low (such as 0-0.2), forming an "edge enhancement mask".

[0128] Fruit center region: Based on the distance distribution of the depth image (if the distance value of the real center pixel is more stable and has a smaller deviation), set the weight value of the fruit center pixel to high (e.g., 0.8-1.0) and the weight value of the edge region to low (e.g., 0-0.2) to form a "center enhancement mask".

[0129] RGB Feature Masking Weighting: Enhancing Fruit Edge Color Texture: The edge enhancement mask is weighted pixel-by-pixel with the RGB color texture features output from the first branch. For example, the RGB features of fruit edge pixels (such as the color gradient of the red edge and the details of the skin texture) are multiplied by a high weight (0.9) and significantly enhanced; while the RGB features of the leaf-occluded area (such as the interference texture of the green leaves) are multiplied by a low weight (0.1) and effectively suppressed. In this way, the RGB features focus on the differences in appearance details at the fruit edge (such as the uniform color of the edge of ripe fruit and the greenish color of the edge of unripe fruit), avoiding background / occlusion interference.

[0130] Reverse masking weighting of NIR features: Enhancing the internal quality of the fruit center: The center enhancement mask (the opposite of the edge enhancement mask, with high weight for the center and low weight for the edges) is weighted pixel-by-pixel with the NIR internal quality features output from the second branch. For example, the NIR features of the fruit center pixels (such as near-infrared reflectance corresponding to sugar content and uniformity of internal maturity) are multiplied by a high weight (0.9) and are retained; while the NIR features of the fruit edges (which are easily affected by skin thickness and light reflection, resulting in low accuracy) are multiplied by a low weight (0.1) and are weakened. This processing allows the NIR features to focus on the "core region of internal quality" in the center of the fruit, ensuring the reliability of the internal features.

[0131] Trimodal feature fusion: The mask-weighted RGB edge features and NIR center features are fused with the distance-geometric features of the third branch. For example, when two overlapping fruits are detected (depth features show that the closer fruit is 500mm away and the farther fruit is 700mm away), the RGB edge features and NIR center features of the closer fruit are retained first, while the interference features of the farther fruit are ignored. Finally, a fused feature vector with three dimensions of appearance edge, internal center and spatial position is generated, which completely solves the feature confusion problem of overlapping / occluded fruits.

[0132] The classifier adds a spatial feature calibration step to the existing appearance-internal feature matching:

[0133] By combining the distance-geometric features in the fusion features, the maturity level is verified a second time. For example, if the RGB edge features show that the color is reddish (similar to ripe), the NIR center features show that the sugar content is up to standard (similar to ripe), but the depth geometric features show that the fruit diameter is only 50mm (not up to ripe size), then the classifier will judge the fruit as an immature small fruit, avoiding misjudgment caused by premature ripening in appearance or failure to meet size standards.

[0134] By incorporating the matching degree of spatial features (such as the similarity between geometric shape and ripe fruit) into the probability calculation, for example, a shape matching degree of ripe fruit ≥0.8 will increase the prediction probability of ripeness level, and vice versa, resulting in a more accurate ripeness level output.

[0135] The effects of the above technical solution are as follows:

[0136] In cold-region greenhouses, vertical cultivation is commonly used, leading to widespread fruit overlap and leaf shading (shading rates can reach over 30%). The original two-branch model was prone to misjudgment due to feature confusion. With the addition of a third branch, a spatial weight mask generated from depth features can accurately distinguish the edges and center areas of overlapping fruits. Even in complex scenarios with a 30% fruit overlap rate and a 25% leaf shading rate, the maturity recognition accuracy remains above 92%, an improvement of 15-20 percentage points compared to the original two-branch model. This completely solves the core pain point of misjudging shaded fruits in cold-region greenhouses.

[0137] By employing a reverse masking strategy that enhances RGB edges and NIR centers, different modal features are focused on their most effective regions: RGB excels at capturing edge details, so edge features are enhanced; NIR excels at reflecting the quality of the center, so center features are enhanced. This avoids the waste of resources that would otherwise be achieved by using RGB to judge the center (low accuracy) or using NIR to judge the edge appearance (low accuracy). Tests show that after masking weighting, the recognition accuracy of RGB edge features is improved by 28%, and the recognition accuracy of NIR center features is improved by 32%, providing a more reliable feature basis for maturity assessment.

[0138] With the addition of the third branch, maturity recognition and spatial positioning are integrated into a single output. The distance-geometric features included in the fusion feature can directly provide the robotic arm with the precise coordinates of the picking target without secondary data processing, thus shortening the response time of the robotic arm from recognition to picking and improving the efficiency of picking in cold greenhouses.

[0139] In one possible implementation, a preprocessing step is included before the image is input into the maturity recognition model:

[0140] The corrected RGB image and the NIR image are enhanced by contrast-limited adaptive histogram equalization (CLAHE), and the RGB image and NIR image are respectively registered with the depth image at the pixel level.

[0141] CLAHE is an improved algorithm based on traditional histogram equalization. Its core principle is to avoid the "local overexposure" problem that traditional algorithms easily cause through "block processing + contrast limiting," thus accurately improving the contrast and detail clarity of cold-region greenhouse images. The specific steps are as follows:

[0142] The corrected RGB and NIR images are divided into multiple equal-sized sub-blocks. Cold-region greenhouse images often have large local contrast differences due to low light and fog (if the brightness difference between the shadow area and the illuminated area is significant). Block processing allows the algorithm to optimize each sub-block individually, avoiding the problem of global equalization ignoring local details.

[0143] For each sub-block, a grayscale histogram is calculated (based on the luminance channel for RGB images and directly on the grayscale value for NIR images), and the number of pixels at each grayscale level is counted. If the number of pixels at a certain grayscale level is too large (for example, low-brightness pixels are concentrated in a foggy image), it will cause overexposure in that area after traditional equalization. Therefore, CLAHE sets a contrast threshold to evenly distribute the number of pixels exceeding the threshold to other grayscale levels, avoiding excessive stretching of local brightness.

[0144] For each sub-block, grayscale mapping is performed based on the adjusted histogram, converting the original grayscale values ​​into new grayscale values ​​and improving the contrast within the sub-block. Since block processing may result in obvious boundaries between sub-blocks, a bilinear interpolation algorithm is used to smooth these boundaries, ensuring a natural brightness transition across the entire image. The final output is an RGB and NIR image with improved contrast and clearer details. CLAHE enhancement is used to address the issue of blurred details in cold-region images and improve feature recognition.

[0145] In cold-region greenhouses, RGB, NIR, and depth images are typically acquired by different sensors (or different modes of the same sensor). This can easily lead to pixel misalignment due to installation position deviations and differences in optical parameters (e.g., the position of a fruit pixel in an RGB image is inconsistent with its corresponding position in a depth image). Pixel-level registration achieves pixel-level alignment of multimodal images through "spatial coordinate mapping." The specific steps are as follows:

[0146] First, the acquisition equipment is calibrated to obtain the intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as positional offset and rotation angle between cameras) of the RGB camera, NIR camera, and depth camera. The low temperature in the cold-region greenhouse environment may cause slight deformation of the sensors, necessitating periodic recalibration to ensure parameter accuracy and provide a basis for coordinate transformation.

[0147] Using the pixel coordinates of the depth image as a reference, the pixel coordinates of the RGB and NIR images are converted into three-dimensional world coordinates. The pixel coordinates of RGB / NIR are converted into ray directions in the camera coordinate system using camera intrinsic parameters. Then, combined with the distance information of the depth image (the actual distance from the pixel to the camera), the three-dimensional world coordinates corresponding to the RGB / NIR pixels are calculated. Subsequently, based on the pixel-world coordinate correspondence of the depth image, the three-dimensional world coordinates of RGB / NIR are reverse-mapped back to the pixel coordinates of the depth image, resulting in a pixel correspondence table between RGB, NIR and depth images.

[0148] Based on the pixel correspondence table, pixel values ​​from the RGB and NIR images are assigned to the corresponding pixel positions in the depth image. If the resolution of the RGB / NIR image differs from that of the depth image, the resolution is adjusted by downsampling (RGB / NIR) or upsampling (depth). Then, missing pixel values ​​are filled in using nearest-neighbor interpolation or bilinear interpolation, ultimately achieving multimodal image alignment for the same pixel position corresponding to the same fruit region. Pixel-level registration solves the multimodal image misalignment problem and ensures the reliability of feature fusion.

[0149] In one possible implementation, a training set is obtained by collecting a large number of fruit images at different maturity stages in orchard environments and accurately labeling them, which is then used for a maturity recognition model. The maturity recognition model is trained using the training set that has undergone data augmentation. The data augmentation process includes applying rotation, scaling, and adding simulated fog and frost noise to the original labeled images to improve the model's generalization ability under different shooting conditions and interference environments.

[0150] Collect and accurately label images of fruits at different maturity stages in orchard environments to ensure that the training set covers common crop varieties (tomatoes, strawberries, cucumbers, etc.), growth status (size and morphological differences), and environmental background (different cultivation racks and leaf density) in cold greenhouses. This allows the model to learn diverse fruit characteristics and environmental information in advance, eliminating the need for additional training for specific scenarios in actual cold greenhouse applications and shortening the scenario adaptation cycle.

[0151] During data augmentation, rotation (random rotation from 0 to 360°) and scaling (random scaling from 0.8 to 1.2 times) operations are used to simulate image distortion caused by camera angle shifts and distance changes during actual shooting. By adding simulated fog and frost noise (simulating the fog and frost coverage effect in a cold greenhouse environment with high humidity and low temperature), the model can adapt to the interference scene of blurred images and weakened features in advance, effectively solving the core problem of model failure caused by environmental interference.

[0152] In one possible implementation, the maturity recognition model is optimized through a combined loss function, which is a weighted sum of the cross-entropy loss function and the focal loss function, to address the model training problem under the condition of imbalanced sample size for maturity categories.

[0153] Cross-entropy loss excels at optimizing for classes with a balanced number of samples, while FocalLoss addresses the problem of insufficient training for minority classes by reducing the weights of easily classified samples (majority class) and increasing the weights of difficult-to-classify samples (minority class). By weighting and fusing the two, the model can ensure recognition accuracy for the majority class (e.g., mature classes) through cross-entropy loss, while strengthening learning for the minority class (e.g., immature and overmature classes) through FocalLoss.

[0154] In one possible implementation, the maturity recognition model incorporates a feature weight adaptive mechanism; the mechanism determines the growth stage of the fruit based on the fruit volume estimated from the depth image; and dynamically adjusts the weights of color texture features and internal quality features in the fused features at different growth stages: assigning higher weights to internal quality features during the young fruit stage and assigning higher weights to color texture features during the mature fruit stage.

[0155] During the young fruit stage, the appearance characteristics of the fruit are not obvious (e.g., all are green with little color difference), but the internal quality (e.g., the rate of sugar accumulation) is the key to judging the growth status. At this time, the model estimates the fruit volume through depth images (young fruit volume is small) and automatically assigns higher weights (e.g., 0.7) to the internal quality features (NIR extraction) to ensure accurate judgment of the growth status of the young fruit. During the mature fruit stage, the appearance characteristics are clear (e.g., significant differences in color and texture), and the model assigns higher weights (e.g., 0.8) to the color and texture features (RGB extraction) to accurately identify the maturity level.

[0156] In one possible implementation, the robotic arm is a flexible robotic arm, and S4 includes:

[0157] Based on data collected by the wide-angle / fisheye camera, LiDAR, and end-effector flexible tactile array mounted on the flexible robotic arm, crops are divided into avoidable targets (such as branches and leaves) and conformable targets (such as slightly bendable stems) through a visual semantic segmentation model. Combined with obstacle depth information generated by LiDAR, a dynamic obstacle map is constructed.

[0158] When planning a path for a flexible robotic arm, the physical deformation constraints and dynamic obstacle constraints of the flexible robotic arm are used together as constraints for path search. The physical deformation constraints include avoidance deformation curvature and avoidance bending deformation angle, and the dynamic obstacle constraints include the minimum distance between the path and the avoidance target not being less than a preset safety threshold and the minimum distance between the path and the compliant target not being less than a preset compliance threshold.

[0159] Among the set of paths that simultaneously satisfy the physical deformation constraints and dynamic obstacle constraints, the optimal path from the starting point of the robotic arm to the target point of the fruit is selected; the optimal path must meet the evaluation criteria of the shortest path length and the lowest deformation energy consumption of the robotic arm.

[0160] The working principle of the above technical solution is as follows:

[0161] Addressing the challenges of dense crop canopies and irregular obstacle distribution in high-density planting spaces, the flexible robotic arm obstacle avoidance design employs a dual-core logic of multi-sensor fusion modeling and deformation-constrained path planning to achieve safe and efficient movement, as detailed below:

[0162] By using three types of sensors—vision, lidar, and tactile—to work together, the robotic arm can fully perceive the environment around it, accurately distinguish the types and distribution of obstacles, and provide data support for obstacle avoidance.

[0163] The panoramic image is processed by a visual semantic segmentation model. Based on crop morphological features (such as leaf texture and stem thickness), the crops in the image are divided into avoidance targets (such as lignified hard branches and key branches and leaves next to mature fruits, which are easy to be damaged by collision) and compliance targets (such as non-lignified soft stems and senescent leaves, which can withstand slight bending without affecting crop growth).

[0164] The semantic segmentation results are then fused with the 3D point cloud data generated by LiDAR. Point clouds that avoid the target are assigned high weights (marked as strongly constrained obstacles), while point clouds that conform to the target are assigned low weights (marked as weakly constrained obstacles). A high-precision dynamic obstacle map containing obstacle type, 3D coordinates, and constraint level is constructed.

[0165] During the movement of the robotic arm, the vision sensor and LiDAR update the data every 0.1 seconds, and synchronously correct the obstacle map (such as the position shift of branches and leaves caused by slight touch of the robotic arm), to ensure that the map matches the actual environment in real time and avoid obstacle avoidance lag caused by static modeling.

[0166] In high-density planting spaces, flexible robotic arms need to move in narrow gaps. Path planning must simultaneously avoid obstacles and not exceed their own deformation limits. The specific steps are as follows:

[0167] Based on the material properties of the flexible robotic arm (such as the bending limit of silicone material and the deformation stiffness of carbon fiber skeleton), two types of core physical deformation constraints are set: ① Avoidance deformation curvature: specify the maximum bending curvature of each joint of the robotic arm during obstacle avoidance to avoid excessive bending that could damage the mechanical structure; ② Avoidance bending deformation angle: limit the bending angle range of each joint to prevent deformation from exceeding the control range of the drive components, which could lead to a decrease in motion accuracy.

[0168] Using the starting position of the robotic arm (such as the initial point on the greenhouse operation track) and the target fruit point (three-dimensional coordinates output by the maturity recognition model) as the path endpoints, multiple potential motion paths are generated in a high-precision dynamic obstacle map by a fast exploration random tree algorithm.

[0169] Deformation constraint verification is performed on each potential path: simulate the deformation state of each joint when the robotic arm moves along the path, calculate the deformation curvature and bending angle of each moving node, eliminate paths that exceed the deformation constraints (e.g., a path requires the elbow joint to bend 100°, and is eliminated if it exceeds the 90° limit), and retain the set of all feasible paths that meet the constraints.

[0170] From the set of feasible paths, the optimal path is selected through a multi-objective evaluation function. Evaluation indicators include: ① Path length: prioritizing shorter paths to reduce movement time (e.g., in high-density spaces, shortening the path by 10% can reduce the picking time per fruit by 0.5 seconds); ② Obstacle distance: ensuring the minimum distance between the path and the "avoidance target" is no less than 5mm, and the minimum distance to the conforming target is no less than 2mm, avoiding collision risks; ③ Deformation energy consumption: selecting the path with the least deformation of the robotic arm (e.g., the smallest total bending angle) to reduce drive energy consumption and structural damage. The final output is the optimal motion path that balances short path, high safety, and low energy consumption.

[0171] In one possible implementation, the path planning employs a spatial partitioning-based algorithm, including:

[0172] Using the set of obstacle spatial points in the pre-constructed 3D point cloud map of the environment as the root node, the workspace of the robotic arm is recursively divided into volumes until the volume of the sub-regions reaches the preset minimum value, forming a discrete, passable path network that satisfies flexible deformation constraints, and then path search is performed in this network.

[0173] The root node is recursively segmented using either a binary or octet method (chosen based on the spatial dimension; the octet method is commonly used in 3D space). Each segmentation round must incorporate obstacle constraints and flexible robotic arm deformation constraints to ensure that the sub-region is passable and does not violate regulations. The specific steps are as follows:

[0174] Determine the segmentation conditions: Calculate the volume of the current area. If the volume is greater than the preset minimum value (e.g., 10cm×10cm×10cm, set according to the size of the crop gap and the size of the robotic arm end effector), and the area contains a set of obstacle points or has potential collision risks, then segmentation is triggered. If the volume has reached the preset minimum value, or there are no obstacles in the area and the robotic arm deformation constraints are fully met (e.g., the bending angle of any path in the area is ≤ the maximum bending angle of the robotic arm), then segmentation stops and the area is marked as a passable sub-area.

[0175] The current region is uniformly divided into multiple sub-regions along the x, y, and z axes (e.g., divided into 8 small cubes using the octet method), and each sub-region is double-checked.

[0176] Check if the sub-region overlaps with the avoidance target point set. If the overlapping area accounts for more than 30% of the sub-region volume (preset threshold), it is marked as an impassable sub-region and removed. If it overlaps with the conforming target point set, and the robotic arm can avoid it by slight deformation (e.g., bending angle ≤ 5°) within the overlapping area, the sub-region is retained and marked as a low-constraint passable sub-region.

[0177] The maximum deformation state of the simulated robotic arm during its movement within a sub-region (e.g., the bending curvature of the robotic arm corresponding to the diagonal length of the sub-region) is considered. If the deformation value (bending curvature, bending angle) does not exceed the preset constraints (e.g., bending curvature ≤ 0.05mm), then... -1 If the curvature angle is ≤90°, then the sub-region is confirmed to meet the passage conditions; if it exceeds the constraints, it is marked as an "inaccessible sub-region" and removed.

[0178] For sub-regions that pass the double verification, repeat the above judgment, segmentation, and verification process until all sub-regions reach the preset minimum volume or are marked as impassable, ultimately forming a discretized three-dimensional spatial structure composed of a large number of passable sub-regions.

[0179] All walkable sub-regions are treated as network nodes. A path network is constructed based on the spatial adjacency between sub-regions (e.g., two sub-regions share a face, edge, or vertex). Connecting edges are established between adjacent walkable sub-regions, and the edge weights are set based on the distance between the sub-region centers and the deformation energy consumption of the robotic arm within the sub-region (the shorter the distance and the lower the energy consumption, the smaller the weight). Then, Dijkstra's algorithm or A* algorithm is used to perform path search within this network.

[0180] The starting position of the robotic arm (e.g., the initial coordinates of the base) and the target position of the fruit (three-dimensional coordinates output by the maturity recognition model) are mapped into the path network respectively, and the starting sub-region containing the starting point and the target sub-region containing the ending point are found.

[0181] Starting from the initial sub-region, traverse the network nodes in ascending order of connected edge weights, calculate the cumulative weight (cumulative distance plus cumulative energy consumption) from the initial sub-region to each sub-region, and prioritize the path with the smallest cumulative weight until the target sub-region is found.

[0182] The searched sub-region sequence is converted into a continuous three-dimensional path. The inflection points of the path are smoothed by Bézier curves to ensure that the deformation is continuous and without abrupt changes during the movement of the robotic arm. Finally, the optimal path that meets the requirements of no collision, low energy consumption and deformation constraints is output.

[0183] Recursive volume segmentation precisely divides obstacles and passable spaces into small sub-regions. Even in high-density scenarios where the crop canopy overlap rate reaches 60%, it can control the minimum distance between passable sub-regions and obstacles to more than 5mm, effectively protecting crops and robotic arm structures.

[0184] Deformation constraint verification is performed simultaneously during the segmentation process to ensure that the paths in all passable sub-regions meet the bending curvature and bending angle limits of the robotic arm. The excessive deformation rate of the robotic arm is reduced from 3% in the traditional algorithm to less than 0.5%, the service life of joint components is extended by more than 30%, and the later maintenance costs are reduced.

[0185] By recursively partitioning the continuous 3D workspace into finite walkable sub-regions, path search no longer needs to randomly sample in an infinite continuous space, thus shortening the search time. The weight settings (distance + energy consumption) of the connected edges in the path network make the search process more targeted, avoiding the problem of traditional algorithms blindly exploring meaningless regions.

[0186] When the greenhouse environment undergoes dynamic changes (such as foliage growth or reduced obstacles after fruit harvesting), it is not necessary to rebuild the entire path network. Only the sub-regions of the changed area need to be re-segmented and validated (e.g., deleting sub-regions containing harvested fruit and adding impassable sub-regions due to foliage extension) to adapt to the dynamic environmental changes caused by crop growth. The preset minimum sub-region volume can be flexibly adjusted according to crop type, without modifying the core algorithm logic, thus adapting to high-density planting scenarios for various crops such as strawberries, tomatoes, and cucumbers in cold-region greenhouses.

[0187] The online fine-tuning specifically includes:

[0188] After each harvesting attempt, the system records the original multispectral data, environmental data, confidence level of the identification results, and final execution results for this operation;

[0189] When a harvest is successful and the recognition confidence level is higher than the first threshold, the data set is determined to be a high-reliability positive sample; when a harvest fails but the recognition confidence level is higher than the second threshold, the data set is determined to be a high-value negative sample; the above samples are stored in a circular cache pool.

[0190] A preset number of samples (small batch samples) are periodically extracted from the cache pool, and the image correction network and maturity recognition model are incrementally trained with a preset learning rate (small learning rate).

[0191] In one possible implementation, the online fine-tuning is modular incremental fine-tuning, specifically including:

[0192] Analysis of the reasons for harvesting failure: If it is due to spatial positioning deviation, then only the depth data of this operation is used to fine-tune the three-dimensional coordinate regression branch in the maturity recognition model; if it is due to image recognition error, then only the dynamic weight prediction module in the image correction network is fine-tuned.

[0193] By using a dual criterion of harvesting results and confidence levels to screen samples, high-reliability positive samples (successful harvests + high confidence levels) strengthen the model's learning of the correlation between correct identification and successful harvesting, while high-value negative samples (failed harvests + high confidence levels) accurately pinpoint the core issues where the model correctly identifies but deviates from the intended purpose (e.g., accurate maturity identification but inaccurate spatial positioning). Both are crucial data for model optimization. Compared to traditional indiscriminate sample collection, this screening method increases the efficiency of training samples by over 60%, avoids redundant data consuming computational resources, and allows the model to focus on optimizing key issues in each training round.

[0194] The recurrent cache pool stores the latest harvested data in real time, ensuring that the model training samples remain synchronized with the current greenhouse environment (such as differences in light caused by seasonal changes and morphological changes caused by crop growth). For example, in cold winters when greenhouse humidity increases, newly collected high-value negative samples will contain data related to image correction bias under high humidity conditions. Through continuous updates in the cache pool, the model can learn parameters to cope with the new environment in a timely manner, avoiding performance degradation due to sample lag.

[0195] Incremental training using small batches of samples (e.g., 16-32 groups each time) and a small learning rate (e.g., 1 / 10-1 / 20 of the base learning rate) allows the model to slowly absorb new sample information while avoiding catastrophic forgetting (forgetting the effective parameters learned in the past) caused by large-scale training or high learning rates.

[0196] Training with small batches of samples requires minimal computation, and the time for a single round of incremental training can be controlled within 10–30 seconds, without interrupting the normal greenhouse harvesting process (typically, a single fruit harvesting cycle is 8–10 seconds, and training can be interspersed between two harvesting intervals). This ensures the continuity of large-scale harvesting in cold-region greenhouses, and the daily harvesting efficiency is not affected by training.

[0197] During the control of the movement of the flexible robotic arm, the contact force with the environment is monitored in real time by a flexible tactile array integrated on the surface of the robotic arm. When an unexpected collision is detected and the contact force exceeds the safety threshold, the control system immediately triggers the compliance mode, controlling the robotic arm to flexibly bend or retract in the opposite direction of the force to avoid obstacles.

[0198] Example 2: This application provides a multispectral visual positioning harvesting system for cold-region greenhouses, used in the multispectral visual positioning harvesting method for cold-region greenhouses described in Example 1. The system includes:

[0199] The multispectral visual perception module includes at least one RGB-NIR multispectral camera, at least one depth camera, and a temperature and humidity sensor, used to simultaneously acquire RGB images, NIR images, depth information, and greenhouse environment temperature and humidity data of the target.

[0200] The control module, with a built-in processor and memory, is used to store and run the image correction network, maturity recognition model, and path planning algorithm, and to generate control commands.

[0201] The execution module includes a multi-degree-of-freedom robotic arm and a harvesting actuator at its end, used to receive the control commands and complete the harvesting action.

[0202] In one possible implementation, the multispectral visual perception module is detachably installed at the end of the flexible actuator, forming an "eye on hand" configuration; the computing and control center has a pre-stored library of model parameters for different crops, and users can call up the corresponding model parameters according to the types of crops grown in the greenhouse.

[0203] The working principle and effect of the above technical solution are the same as those in the method embodiments of this application, and will not be repeated here.

[0204] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described in this application.

[0205] This application also provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it implements the steps of any of the methods described in this application. The specific implementation method is consistent with the implementation method and the technical effect achieved in the above method embodiments, and some contents will not be repeated.

[0206] In this application, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program product can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0207] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof. Program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on a user computing device, partially on an associated device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0208] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the invention without departing from the principles and spirit of the invention, and all such changes should fall within the protection scope of the claims of the present invention.

Claims

1. A multispectral visual positioning harvesting method for cold-region greenhouses, characterized in that, The method includes: S1: Simultaneously acquire RGB images, NIR images, greenhouse environment temperature and humidity data, and depth information of the target; S2: Based on the RGB image, NIR image and environmental data, image correction is performed through an image correction network to output a sharpened RGB image; the image correction network dynamically modulates the feature fusion weights of RGB and NIR through environmental prior vectors, and its training process is constrained by the atmospheric scattering physics model; S3: Input the corrected RGB image, NIR image and depth image into the maturity recognition model for processing. The maturity recognition model extracts and fuses color, texture and internal quality features through a cross-modal attention fusion network, and outputs the maturity level and three-dimensional spatial coordinates of the fruit. S4: Based on the three-dimensional coordinates of the fruit, maturity information, and a pre-constructed three-dimensional point cloud map of the environment, plan the optimal picking path for the robotic arm without collision. S5: Control the robotic arm to perform the picking action, and use the success rate and recognition confidence of the picking results as feedback signals to fine-tune the parameters of the image correction network and maturity recognition model online.

2. The multispectral visual positioning and picking method according to claim 1, characterized in that, S2 includes: S21. Read the ambient temperature and humidity sensor data and calculate the sharpness evaluation index of the RGB image to form an environmental prior vector. S22: Input the environmental prior vector into the parameter prediction network to generate a dynamic weight matrix corresponding to RGB features and NIR features; use the weight matrix to perform weighted fusion of the extracted RGB color texture features and NIR structure edge features; generate a preliminary fused feature map; S23: Input the preliminary fused feature map into the generator network to generate the corrected image.

3. The multispectral visual positioning and picking method according to claim 2, characterized in that, The generator is optimized by fusing a loss function, which includes adversarial loss, perceptual loss, and physical consistency loss. The physical consistency loss term is constructed based on an atmospheric scattering model and is used to constrain the relationship between the generated image, the input image, the estimated transmittance map, and atmospheric light.

4. The multispectral visual positioning and picking method according to claim 2, characterized in that, The sharpness evaluation index is the average image gradient and / or image entropy; The parameter prediction network is a multilayer perceptron; The dynamic weight matrix is ​​either a channel-level weight vector or a spatial-level attention map.

5. The multispectral visual positioning and picking method according to claim 1, characterized in that, The maturity identification model is a multi-branch network architecture, including: The first encoding branch is used to extract color and texture features from the corrected RGB image; The second encoding branch is used to extract internal quality features from the NIR image; A cross-modal attention fusion module is used to interact and fuse features extracted from multiple branches; the cross-modal attention fusion module is configured to enable features of any modality to be used as a query vector to interact with features of another modality in order to retrieve and focus on complementary information. A classifier is used to output a maturity level based on the fused features.

6. The multispectral visual positioning and picking method according to claim 5, characterized in that, The maturity recognition model also includes a third coding branch for extracting spatial distance and geometric features from the depth image; The cross-modal attention fusion module is configured to: use the distance features extracted by the third coding branch as a shared query vector to guide and modulate the attention of the first coding branch to the color and texture features of the fruit edge, and the attention of the second coding branch to the quality features inside the center of the fruit. The execution process of the cross-modal attention fusion module includes: generating a spatial weight mask based on depth features, applying mask weighting to RGB features to enhance the response of the fruit edge region, and applying reverse mask weighting to NIR features to enhance the response of the fruit center region.

7. The multispectral visual positioning and picking method according to claim 1, characterized in that, Before inputting the image into the maturity recognition model, a preprocessing step is also included: The corrected RGB image and the NIR image are enhanced with contrast-limited adaptive histogram equalization, and the RGB image and the NIR image are respectively registered with the depth image at the pixel level.

8. The multispectral visual positioning and picking method according to claim 1, characterized in that, The robotic arm is a flexible robotic arm, and S4 includes: When planning a path for a flexible robotic arm, the physical deformation constraints of the flexible robotic arm are used as constraints for path search; the physical deformation constraints include avoidance deformation curvature and avoidance bending deformation angle. From the set of paths that satisfy the constraints, select the optimal path from the starting point of the robotic arm to the target point of the fruit.

9. The multispectral visual positioning and picking method according to claim 1, characterized in that, The path planning employs a spatial partitioning-based algorithm, including: Using the set of obstacle spatial points in the pre-constructed 3D point cloud map of the environment as the root node, the workspace of the robotic arm is recursively divided into volumes until the volume of the sub-regions reaches the preset minimum value, forming a discrete, passable path network that satisfies flexible deformation constraints, and then path search is performed in this network.

10. A multispectral visual positioning harvesting system for cold-region greenhouses, used to implement the multispectral visual positioning harvesting method for cold-region greenhouses as described in any one of claims 1-9, characterized in that, The system includes: The multispectral visual perception module includes at least one RGB-NIR multispectral camera, at least one depth camera, and a temperature and humidity sensor, used to simultaneously acquire RGB images, NIR images, depth information, and greenhouse environment temperature and humidity data of the target. The control module, with a built-in processor and memory, is used to store and run the image correction network, maturity recognition model, and path planning algorithm, and to generate control commands. The execution module includes a multi-degree-of-freedom robotic arm and a harvesting actuator at its end, used to receive the control commands and complete the harvesting action.

Images