Greenhouse fruit picking method and device based on digital twinborn model

Abstract

The invention provides a greenhouse fruit picking method based on a digital twinborn model, which comprises the following steps: acquiring basic data of a greenhouse and attribute data and growth data of crops in the greenhouse, and constructing the digital twinborn model of the crops based on the basic data, the attribute data and the growth data; on the basis of the digital twinborn model, current ripening information of the fruits is obtained, and a picking scheme is generated through the current ripening information; picking operation is performed according to the picking scheme; predicting future ripening information of the fruits based on the digital twinborn model, and generating a picking plan through the future ripening information; and performing picking preparation work based on the picking plan.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a greenhouse fruit picking method and device based on a digital twin model. Background Art

[0002] Existing tomato-picking robots typically capture and harvest tomato in one area before moving to the next area to capture and harvest tomato. This localized picking model lacks a global perspective and cannot capture comprehensive information about the orchard or greenhouse, making it difficult to optimize the picking path globally and significantly reducing picking efficiency.

[0003] Existing vision technology primarily relies on a two-step strategy of "detection-maturity determination." This can only determine the maturity of the fruit in its current state and cannot predict its future maturity. This limitation leads to multiple problems in the picking process. On the one hand, the picking timing is difficult to optimize, and the picking plan cannot be dynamically adjusted based on the number of ripe fruits. For example, if the number of ripe fruits is too large, some may overripe, fall, or even rot due to untimely picking. If the number of ripe fruits is too small, the input-output ratio of picking is not high. On the other hand, the robot may waste time on unripe tomatoes during the picking process, further increasing the time cost and resource consumption of picking.

[0004] Previous tomato-picking robots typically employed an "eyes in hands" operating mode. The main process can be summarized as follows: First, the robot's robotic arm, equipped with a visual sensor (depth camera), captures localized images within a target area. After capturing the image of the localized area, the robot records the location and ripeness of the fruit. Ripe fruits are harvested sequentially (e.g., from left to right). In recent years, some work has performed simple local planning of multiple ripe tomatoes within its field of view to optimize the picking order. However, the number of tomatoes within a local field of view is typically limited to a few. After harvesting the current area, the robot moves to the next area and repeats the image capture and data collection process until all areas within its operating range are covered. Because the robotic arm is often very close to the tomato plant, it is difficult for the camera to cover the entire plant in front of it. In addition to scanning from side to side, the robotic arm also scans the tomatoes from top to bottom.

[0005] The entire process relies solely on local information, failing to optimize picking paths or dynamically adjust picking strategies from a global perspective. This, coupled with the need for multiple inspections, results in low picking efficiency and high time costs, making it difficult to meet the practical application needs of the complex greenhouse environment. Furthermore, the fruit's condition is determined only by current static information, lacking the ability to predict future ripening, making it difficult to guide picking priority planning and timing.

[0006] Local field of view limitations: Current methods rely on robotic arms carrying visual sensors to capture images of local areas, failing to obtain global information about the greenhouse or orchard, limiting the global optimization of the picking path.

[0007] The picking strategy lacks global planning: Although some methods attempt to perform simple planning for ripe tomatoes within a local field of view, the overall picking path fails to be optimized from a global perspective, resulting in low picking efficiency.

[0008] Limited scanning angle of the robotic arm: Due to the close distance between the robotic arm and the tomato plant, the camera cannot fully cover the entire area of the tomato plant. During the picking process, the robotic arm still needs to move up and down to scan, which increases the complexity and time cost of the operation.

[0009] Multiple inspections and repetitive operations: The robot needs to conduct multiple inspections and move to different areas for picking, which makes the picking process repetitive and time-consuming, increasing time costs and resource consumption.

[0010] Lack of future maturity prediction: Existing methods rely solely on the current maturity of the fruit and are unable to predict future maturity. This makes it difficult to optimize picking timing and priority, potentially leading to resource waste or overripe and rotten fruit.

[0011] In summary, existing technologies have obvious shortcomings in terms of picking path optimization, picking time selection and resource utilization efficiency, and more forward-looking solutions are urgently needed. Summary of the Invention

[0012] In response to the above problems, the present invention proposes a greenhouse fruit picking method based on a digital twin model, comprising: collecting basic data of the greenhouse, and attribute data and growth data of the crops in the greenhouse, and constructing a digital twin model of the crops based on the basic data, the attribute data and the growth data; the basic data includes the spatial dimensions, planting layout and environmental data of the greenhouse, the attribute data includes the external dimensions, color texture and picking requirements of the fruits of the crops in each growth period, and the growth data includes the current external dimensions and color texture of the fruits, and the current external dimensions of the crops; based on the digital twin model, the current maturity information of the fruit is obtained, and a current picking plan is generated according to the current maturity information; the current picking plan is used to drive a picking robot to perform picking operations; based on the digital twin model, the maturity trend information of the fruit is predicted, and a picking plan for future picking operations is generated according to the maturity trend information; and future picking preparations are carried out based on the picking plan.

[0013] Furthermore, the step of generating a picking plan includes: obtaining historical image data of fruits in each growth period to generate a maturity training set, training a reinforcement learning network, and obtaining a maturity judgment model; the maturity judgment model generates the current maturity information, and through the current maturity information, all fruits are identified as mature fruits and unripe fruits; a picking training set is generated based on the historical picking records of the crop, and a reinforcement learning network is trained to obtain a picking plan generation model; and the current picking plan is generated through the picking plan generation model.

[0014] Furthermore, the step of generating a picking model includes: based on the digital twin model, 3D modeling of all the fruits to generate a picking model, which includes the plant dimensions of the crop where the fruit is located and the spatial position of the fruit; based on the picking plan generation model and the picking model, the current picking plan is generated.

[0015] Furthermore, the step of generating a harvesting plan includes: training a reinforcement learning network with the maturity training set to obtain a maturity trend prediction model; and the step of preparing for future harvesting includes:

[0016] Based on the digital twin model and the maturity trend prediction model, the maturity period of the immature fruit is predicted, and based on the maturity period, a picking plan for the immature fruit is generated; before the fruit is picked, the picking plan is adjusted according to the fruit picking demand.

[0017] The present invention also proposes a greenhouse fruit picking device based on a digital twin model, comprising: a model construction module for constructing a digital twin model of the crop; including collecting basic data of the greenhouse, and attribute data and growth data of the crops in the greenhouse, and constructing the digital twin model based on the basic data, the attribute data and the growth data; the basic data includes the internal space size and planting layout of the greenhouse, the attribute data includes the external dimensions, color texture and picking requirements of the fruits of the crop in each growth period, and the growth data includes the current external dimensions and color texture of the fruit, and the current external dimensions of the crop plant; a picking module for obtaining the current maturity information of the fruit based on the digital twin model, generating a current picking plan through the current maturity information; driving a picking robot to perform picking operations with the current picking plan; a prediction module for predicting the maturity trend information of the fruit based on the digital twin model, generating a picking plan for future picking operations through the maturity trend information; and performing future picking preparations based on the picking plan.

[0018] Furthermore, the picking module includes: a maturity model training module, which is used to obtain historical image data of fruits in each growth period to generate a maturity training set, train the reinforcement learning network, and obtain a maturity judgment model; the maturity judgment model generates the current maturity information, and through the current maturity information, all fruits are identified as mature fruits and unripe fruits; a picking model training module, which is used to generate a picking training set based on the historical picking records of the crop, train the reinforcement learning network, and obtain a picking plan generation model; the current picking plan is generated through the picking plan generation model.

[0019] Furthermore, the picking model training module includes: a 3D modeling module, which is used to perform 3D modeling of all the fruits based on the digital twin model to generate a picking model, wherein the picking model includes the plant dimensions of the crop where the fruit is located and the spatial position of the fruit; based on the picking plan generation model and the picking model, the current picking plan is generated.

[0020] Furthermore, the prediction module includes: a trend model training module, which trains the reinforcement learning network with the maturity training set to obtain a maturity trend prediction model; a plan generation module, which is used to predict the maturity period of the immature fruit based on the digital twin model and the maturity trend prediction model, and generate a picking plan for the immature fruit based on the maturity period; and a plan adjustment module, which is used to adjust the picking plan according to the fruit picking needs before the fruit is picked.

[0021] The present invention also proposes an electronic device, including the greenhouse fruit picking device based on the digital twin model as described above.

[0022] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the greenhouse fruit picking method based on the digital twin model as described above is implemented.

[0023] The present invention aims to significantly improve existing problems such as insufficient picking path optimization, inaccurate timing selection, complex robotic arm operation, and poor environmental adaptability by combining a digital twin model with an intelligent decision-making method, thereby providing an efficient, intelligent, and economical solution for tomato-picking robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the greenhouse fruit picking method based on the digital twin model of the present invention.

[0025] Figure 2 It is a flow chart of the data acquisition process of the present invention.

[0026] Figure 3 It is a schematic diagram of the dynamic collection of crop growth data of the present invention.

[0027] Figure 4 It is a flow chart of the picking decision generation process of the present invention.

[0028] Figure 5 It is a schematic diagram of picking decision generation of the present invention.

[0029] Figure 6 It is a flow chart of the generation process of the present invention.

[0030] Figure 7 It is a diagram of the slidable camera system of the present invention.

[0031] Figure 8A This is a flow chart of greenhouse tomato picking according to the present invention.

[0032] Figure 8B It is a schematic diagram of greenhouse tomato picking according to the present invention.

[0033] Figure 9 It is a schematic diagram of the greenhouse fruit picking device based on the digital twin model of the present invention.

[0034] Figure 10 It is a schematic diagram of the model building module structure of the present invention.

[0035] Figure 11 It is a structural schematic diagram of the picking module of the present invention.

[0036] Figure 12 It is a schematic diagram of the prediction module structure of the present invention.

[0037] Figure 13 It is a schematic diagram of an electronic device of the present invention.

[0038] Figure 14 It is a schematic diagram of the hardware structure of an electronic device of the present invention.

[0039] Wherein, the accompanying drawings are marked as follows:

[0040] 100: Electronic equipment 10: Greenhouse fruit picking device

[0041] 11: Model building module 111: Data acquisition module

[0042] 112: Twin Model Building Module 12: Picking Module

[0043] 121: Maturity model training module 122: Picking model training module

[0044] 1221: 3D Modeling Module 13: Prediction Module

[0045] 131: Trend model training module 132: Plan generation module

[0046] 133: Plan Adjustment Module

[0047] S1, S11, S12, S2, S3, S31, S32, S33, S4, S41, S42, S43: Steps DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0050] The digital twin model is a virtual representation used to accurately reflect the state and behavior of physical objects or systems. By constructing the association between virtual objects in the virtual space and physical objects in the physical space, it realizes the digital mapping of virtual objects to physical objects. It can reflect the appearance, geometry, motion structure, geometric association and other properties of physical objects, and then simulate and predict the current status and development trend of physical objects.

[0051] The main purpose of this invention is to address the shortcomings of existing tomato picking robots in path optimization, picking strategies, global perspective construction and timing selection, and to provide an intelligent decision-making method based on a digital twin model to comprehensively improve picking efficiency, reduce time costs and resource consumption, and reduce fruit loss.

[0052] This paper proposes an intelligent decision-making method based on a digital twin model to optimize robotic tomato harvesting in greenhouse environments. The system uses visual algorithms to scan the greenhouse and the plants and fruit within it, constructing a digital twin model of the greenhouse crops. This digital twin model accurately monitors the maturity and position of the fruit and simulates the future ripening process. Based on the simulation results, a reinforcement learning model is trained to dynamically optimize the harvesting strategy, including the robot's parking position, the robotic arm's picking path, the harvesting time, and the harvesting priority of each row. This method significantly improves harvesting efficiency and reduces plant damage and fruit loss.

[0053] First, by constructing a digital twin model of greenhouse crops, this invention overcomes the limitations of existing localized harvesting methods. This provides the robot with a global perspective, enabling precise analysis of the crop plants throughout the greenhouse and global planning of harvesting paths. This improvement effectively optimizes harvesting sequences and movement paths, reduces repetitive operations, and significantly improves harvesting efficiency.

[0054] Secondly, by combining dynamic monitoring of fruit ripening with predictions of future ripening, this invention overcomes the problem of traditional methods relying solely on current fruit state. This allows the robot to rationally schedule picking times and priorities based on global information and predictions. This not only avoids overripe fruit loss due to late picking, but also improves picking efficiency and resource utilization.

[0055] Thirdly, the present invention further optimizes the operation strategy of the robotic arm, and uses the current picking model constructed by the digital twin model to plan the movement trajectory of the robotic arm in advance, avoiding frequent adjustments to the robotic arm position due to insufficient viewing angle, thereby reducing operation complexity and time cost, while reducing potential damage to plants.

[0056] In addition, the present invention introduces a reinforcement learning model to optimize picking decisions. The training of the picking decision model is completed in the current picking model generated based on the digital twin model. Compared with the training method based on simple simulation data, the training effect of the present invention is significantly better. This simulation environment that restores the real scene can more accurately reflect the complex structure and dynamic lighting conditions in the greenhouse, making the robot's decision-making in picking tasks more accurate. Specifically, the present invention can more accurately plan the robot's parking position and the movement path of the robotic arm, and dynamically control the picking start time and fruit picking priority, thereby significantly improving picking efficiency and decision-making robustness, and further reducing time costs and resource waste.

[0057] Figure 1 This is a flow chart of the greenhouse fruit picking method based on the digital twin model of the present invention. Figure 1 As shown, in a first embodiment of the present invention, a greenhouse fruit picking method is proposed, comprising:

[0058] Step S1, data collection: collecting basic data of the greenhouse, and attribute data and growth data of crops in the greenhouse; Figure 2 As shown, including:

[0059] Step S11: The greenhouse's basic data includes the greenhouse's spatial dimensions, planting layout, and environmental data. The crop attribute data includes the dimensions, color, texture, and harvesting requirements of greenhouse crop fruits at different growth stages. The above data can be obtained by querying data, and the environmental data can also be collected in real time by sensors.

[0060] In step S12, the RGBD depth camera on the automatic slide dynamically collects RGB and depth image data of greenhouse plants to ensure that the overall information of the plants is covered from multiple perspectives, and the growth data of the crops is obtained, including the current size, color, texture and maturity of the fruits, as well as the current size of the crops. Then, based on deep learning, 3D spatial fruit recognition and positioning are performed, and models such as Mask R-CNN are used to detect the maturity (e.g., based on color, texture or spectral characteristics) and spatial position of the fruits. Figure 3 As shown;

[0061] Step S2: Build a digital twin model; build a digital twin model of greenhouse crops based on the collected basic data, attribute data and growth data

[0062] Step S3, generating a picking decision; based on the digital twin model of greenhouse crops, obtaining the current maturity information of the fruit, generating a picking plan based on the current maturity information, and performing the picking operation; Figure 4 As shown, including:

[0063] Step S31, obtain historical image data of fruits in each growth period to generate a maturity training set, train the reinforcement learning network, and obtain a maturity judgment model; the maturity judgment model generates the current maturity information of the fruit, and based on the current maturity information of the fruit, identifies all fruits as mature fruits and unripe fruits.

[0064] In step S32, a picking training set is generated based on the historical picking records of the fruit, and the reinforcement learning network is trained to obtain a picking plan generation model. The picking plan generation model is used to generate a picking plan, which includes a local picking plan for a single crop, a global picking plan for the greenhouse, and a picking action plan for the picking robot. The reinforcement learning picking strategy training uses the picking model generated by the twin model and the DDPG (deep deterministic policy gradient) reinforcement learning algorithm to train the tomato picking robot to optimize its parking position decision and local path planning, thereby improving picking efficiency and reducing the robot arm's movement time and the number of collisions. Figure 5 As shown, including:

[0065] 1. Harvesting path optimization based on deep deterministic policy gradient reinforcement learning (DDPG) trained in a greenhouse with a harvesting model reconstruction: Using a Markov decision process (MDP), the harvesting path and robotic arm motions are trained in a virtual greenhouse harvesting model to reduce path redundancy and the number of stops, thereby optimizing the global harvesting strategy. The harvesting model scenario includes a tomato plant model, fruit distribution, robotic arm, and scene physical characteristics. The environmental state of the harvesting model includes the current position of the robotic arm, the 3D position of the fruit, plant structure information, and surrounding environmental constraints.

[0066] 2. A deep neural network architecture combined with a real-time reward mechanism: This uses an objective function to define picking efficiency (e.g., shortest path, minimal collisions, maximum fruit coverage) as the optimization goal. This uses virtual environment feedback to reinforce the learning model's actions, including collision detection, path optimization, and efficiency evaluation.

[0067] 3. Post-generation integrated picking decision: Provides four decision-making processes based on reinforcement learning training results: local ripening priority decision-making, robot parking location planning, global picking optimization path, and automatic / manual picking mode switching;

[0068] 4. Generate joint decisions on picking timing and path: Taking into account the fruit ripening time and the robot arm's movement path, the picking strategy is dynamically adjusted to ensure the optimal solution for picking timeliness and resource utilization.

[0069] 5. Generate a collision detection strategy for the robot's motion path: Based on the simulation environment and deep learning segmentation results, detect potential occlusions and obstacles in the robot's path and adjust the picking path in real time to avoid collisions.

[0070] Step S33: Implement picking operations based on the picking plan.

[0071] Step S4: Based on the digital twin model of greenhouse crops and the maturity judgment model, predict the future maturity information of the fruit, generate a picking plan based on the future maturity information, and perform picking preparations based on the picking plan; Figure 6 As shown, including:

[0072] Step S41, training the reinforcement learning network with the maturity training set obtained in step S31 to obtain a maturity trend prediction model;

[0073] Step S42: Based on the digital twin model and the ripening trend prediction model, an optimal picking window for the immature fruit is predicted, where the optimal picking window includes the ripening period and optimal picking period of the immature fruit; based on the optimal picking window, a picking plan for the immature fruit is generated, where the picking plan includes a local picking plan for the immature fruit of one or more crops, and an overall picking plan for the greenhouse;

[0074] Step S43: Before the immature fruits are picked, the local picking plan and the overall picking plan are adjusted in real time according to the fruit picking demand. The fruit picking demand here refers to, for example, the user's demand for the maturity of the fruit.

[0075] To improve the accuracy of 3D reconstruction of tomato plants and the efficiency of harvest planning, this paper designed a sliding camera system based on automated slides to dynamically capture RGB and depth information of the plants from low, medium, and high angles. This system aims to achieve high-precision data acquisition and real-time visualization, supporting subsequent simulation modeling and reinforcement learning training. It includes:

[0076] 1. Hardware components, such as Figure 7 shown

[0077] 1.1 Global Camera: Equipped with an Intel D435 depth camera, it supports simultaneous acquisition of high-precision RGB images and depth information, and can obtain reliable data under complex lighting conditions.

[0078] 1.2 Automatic slide rail: An electric slide rail that can slide up and down in the vertical direction, supporting data collection at three viewing angles: low (0.5m), medium (1.2m), and high (1.8m), covering the entire plant.

[0079] 1.3 Scan visualization screen: displays the scanning status and 3D point cloud data in real time, allowing operators to monitor the acquisition effect and adjust parameters in a timely manner.

[0080] 1.4 Power box: provides stable power supply and control module to support long-term operation requirements.

[0081] 2. Software Support

[0082] 2.1 Scanning control program: Automated slide control algorithm, setting fixed intervals and scanning speeds to achieve multi-layer seamless data acquisition.

[0083] 2.2 Data processing and display: In conjunction with the real-time point cloud generation module of the acquisition system, the depth data captured by the camera is converted into a three-dimensional point cloud, and the plant structure is displayed in real time on the visualization screen.

[0084] 2.3 Post-processing and storage: Save the collected data in a point cloud file format (such as .ply) to support subsequent model reconstruction and analysis.

[0085] 3. Collection Process

[0086] 3.1 Initial calibration: Position the slide system in the greenhouse collection area, adjust the camera to focus on the tomato plants, and set the starting and ending heights of the slide.

[0087] 3.2 Multi-view scanning: The camera moves up and down along the slide rail, collecting RGB and depth data at different heights of the plant layer by layer to ensure complete coverage of fruits, leaves and stems.

[0088] 3.3 Real-time visual monitoring: During the acquisition process, the data quality can be checked in real time by scanning the screen. If any missed data or abnormal data is found, the acquisition parameters can be adjusted immediately.

[0089] 3.4 Data storage and backup: All collected data will be automatically classified and stored, and data verification will be performed after collection to ensure file integrity.

[0090] 4. Performance and advantages

[0091] 4.1 Panoramic coverage and high-precision acquisition; the automatic slide supports multi-level perspective acquisition, achieving complete coverage of the entire plant; the high-precision depth camera ensures the capture of detailed data in the complex lighting environment of the greenhouse.

[0092] 4.2 Real-time feedback improves efficiency; visual display of scanning status and point cloud data allows operators to monitor in real time and optimize the acquisition process.

[0093] 4.3 Combining automation and flexibility; the slide rail system can preset parameters to complete fully automated data collection and support adaptation to different greenhouse structures and plant layouts; the system design is flexible and the collection height and camera angle can be adjusted according to the needs of different crops.

[0094] 5. Output and Application

[0095] 5.1 High-quality point cloud data: supports subsequent crop plant model reconstruction and precise fruit positioning.

[0096] 5.2 Simulation modeling support: Provide accurate geometric data foundation for the construction of virtual greenhouse scenes and improve simulation modeling effects.

[0097] 5.3 Picking planning optimization: Picking path planning based on complete plant data improves picking efficiency and reduces robot arm movement waste.

[0098] This rail-mounted camera scanning system, through its highly integrated hardware design and intelligent software control, enables efficient crop plant data collection, providing strong support for subsequent simulation analysis and robotic harvesting tasks. In actual deployment, the system has demonstrated exceptional flexibility and applicability, and can be extended to a wider range of greenhouse crop research and application scenarios.

[0099] Taking the picking of greenhouse tomatoes as an example, as shown in FIG8 , the digital twin greenhouse picking method based on robot scanning and sensor monitoring of the present invention is introduced in detail.

[0100] This implementation aims to dynamically reconstruct a digital twin greenhouse by combining 3D data from scanned tomatoes and plants with real-time greenhouse environmental data. By integrating 3D crop scans captured by a robotic camera with environmental data from various sensors within the greenhouse, a dynamic virtual model of the greenhouse crops and environment is constructed in real time, optimizing the tomato picking decision process. This solution not only improves the real-time and accuracy of picking decisions but also provides a feasible framework for the future development of digital twin models for other crops.

[0101] (1) Input:

[0102] 1. Crop data (scanned data of tomatoes and plants):

[0103] - The robot's onboard cameras (RGB and depth cameras) scan the tomatoes and their plants, collecting 3D images and depth data.

[0104] - The data includes the geometric shape, spatial position, size and structural information of the tomato plant.

[0105] 2. Greenhouse environment data (sensor data):

[0106] - Various sensors arranged in the greenhouse collect environmental information in real time:

[0107] - Temperature sensor: real-time monitoring of the temperature in the greenhouse.

[0108] - Soil moisture sensor: collects soil moisture data.

[0109] - Ambient humidity sensor: monitors air humidity.

[0110] - Light sensor: measures the light intensity in the greenhouse.

[0111] (2) Process:

[0112] 1. Data Collection:

[0113] - The robot uses its onboard camera to scan tomatoes and plants, obtain three-dimensional data, and generate spatial information of the tomatoes and plants.

[0114] - At the same time, sensors in the greenhouse collect environmental data in real time, covering temperature, humidity, soil moisture, light, etc.

[0115] 2. Construction of digital twin greenhouse model:

[0116] - The tomato data obtained by the robot scans is combined with environmental data collected by sensors to construct a digital twin greenhouse model. This model not only statically represents the spatial position of tomatoes and plants, but also dynamically reflects the impact of environmental factors on crop growth and crop growth trends.

[0117] 3. Dynamic simulation and decision support:

[0118] - Based on the digital twin model updated by real-time sensor data, the system will adjust the crop growth model and picking strategy in real time to respond to environmental changes in the greenhouse.

[0119] - The system simulates environmental changes and crop conditions to dynamically adjust picking paths and times, optimizing picking efficiency and resource utilization.

[0120] (3) Output:

[0121] - Dynamic updates of environment and crop status: The digital twin greenhouse model is continuously updated to reflect the crop growth status and environmental conditions in the greenhouse in real time.

[0122] (IV) Expected results:

[0123] 1. Accurate and dynamic decision-making: By combining real-time environmental sensor data with crop data, the digital twin greenhouse can reflect crop growth and environmental changes in real time, optimizing harvesting decisions.

[0124] 2. Dynamic adjustment and strong adaptability: Changes in the greenhouse environment (such as light, humidity, temperature, etc.) can be reflected in the model immediately, adjusting the picking path and decision-making, avoiding the lag of traditional static models.

[0125] 3. Strong scalability: This solution is not only suitable for tomato picking, but the model can also be adjusted according to demand to adapt to the construction of digital twins of other crops, providing technical support for a wider range of agricultural applications.

[0126] This example focuses on the core technology of dynamically reconstructing a digital twin greenhouse, combining scanning and sensor data to highlight its innovativeness and dynamic decision-making capabilities.

[0127] It should be noted that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned steps does not mean the order of execution. The order of execution of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0128] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0129] Figure 9 This is a schematic diagram of the greenhouse fruit picking device based on the digital twin model of the present invention. Figure 9As shown, in a second embodiment of the present invention, a greenhouse fruit picking device 10 is provided, comprising:

[0130] The model building module 11 is used to build a digital twin model of greenhouse crops; Figure 10 As shown, including:

[0131] Data collection module 111 is used to collect basic greenhouse data, as well as attribute data and growth data of crops in the greenhouse. Basic data includes the internal space dimensions and planting layout of the greenhouse. Attribute data includes the external dimensions, color, texture, and picking requirements of greenhouse crop fruits at different growth stages. Growth data includes the current external dimensions, color, texture, and current plant dimensions of the crops.

[0132] A twin model construction module 112 is used to construct a digital twin model of greenhouse crops based on the basic data, attribute data, and growth data collected by the data collection module 111;

[0133] The picking module 12 is used to obtain the current maturity information of the greenhouse fruit based on the digital twin model constructed by the twin model construction module 112, and generate a current picking plan based on the current maturity information; the current picking plan is used to drive the picking robot to perform the picking operation; Figure 11 As shown, including:

[0134] The maturity model training module 121 is used to obtain historical image data of fruits at various growth stages to generate a maturity training set, train a reinforcement learning network, and obtain a maturity judgment model; the maturity judgment model generates the current maturity information, and based on the current maturity information, identifies all fruits as mature fruits or unripe fruits;

[0135] The picking model training module 122 is used to generate a picking training set based on the historical picking records of the crop, train the reinforcement learning network, and obtain a picking plan generation model; and generate the current picking plan through the picking plan generation model;

[0136] The 3D modeling module 1221 is used to perform 3D modeling of all the fruits based on the digital twin model and generate a picking model. The picking model includes the plant dimensions of the crop where the fruit is located and the spatial position of the fruit; based on the picking plan generation model and the picking model, the current picking plan is generated.

[0137] The prediction module 13 is used to predict the ripening trend information of the unpicked fruits in the greenhouse based on the digital twin model constructed by the twin model construction module 112, and generate a picking plan for future picking operations based on the ripening trend information; prepare for future picking based on the picking plan; Figure 12 As shown, including:

[0138] The trend model training module 131 trains the reinforcement learning network using the maturity training set to obtain a maturity trend prediction model;

[0139] A plan generation module 132 is configured to predict the maturity period of the immature fruit based on the digital twin model and the maturity trend prediction model, and generate a plan for picking the immature fruit based on the maturity period;

[0140] The plan adjustment module 133 is used to adjust the picking plan according to the fruit picking demand before the fruit is picked.

[0141] In a third embodiment of the present invention, a computer-readable storage medium is provided. If the functions of the greenhouse fruit picking device based on a digital twin model of the present invention are implemented as software functional units and sold or used as independent products, they can be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This software product, stored on a computer-readable storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the methods described in various embodiments of the present invention. Therefore, in a third embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program for a greenhouse fruit picking method based on a digital twin model. It should be understood that the computer-readable storage medium in this embodiment of the present invention can be volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0142] Figure 13 Schematic diagram of an electronic device of the present invention. Figure 13As shown, in the fourth embodiment of the present invention, an electronic device 100 is proposed, including a greenhouse fruit picking device based on a digital twin model as described above. A person of ordinary skill in the art will understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software functional module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any specific form of combination of hardware and software.

[0143] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0144] The electronic device of the present invention may be any device with data processing capabilities, such as a computer or other device. Device embodiments may be implemented through software, hardware, or a combination of software and hardware. For example, a device implemented in software, as a logical device, is formed by a processor of any device with data processing capabilities reading corresponding computer program instructions from a non-volatile memory into memory and executing them. Figure 14 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Figure 14 As shown in the figure, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the greenhouse fruit picking device based on the digital twin model of the present invention is located. Figure 14 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0145] When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0146] This invention combines RGBD depth information acquisition, 3D greenhouse reconstruction technology, reinforcement learning algorithms, and a picking decision optimization model to provide a complete intelligent picking solution for tomato picking robots. Compared with existing technologies, it has the following significant benefits:

[0147] 1. Improve picking efficiency

[0148] Harvesting time has increased by over 30%. Deep learning is used to identify ripe tomatoes, and reinforcement learning is used to optimize the picking path, significantly shortening the time it takes for the robot to perform each picking action. Furthermore, by reducing the number of stops the robot makes in the greenhouse, global path optimization is achieved, further improving overall harvesting efficiency.

[0149] 2. Reduce the distance the robot moves

[0150] The robotic arm's travel distance has been reduced by over 20%. A reinforcement learning model trained in a virtual simulation environment simulates picking paths and movement details in advance, effectively reducing redundant robotic arm movements. The picking decision system selects the optimal picking sequence based on fruit distribution and occlusion, avoiding repetitive robotic arm movements.

[0151] 3. Reduce the risk of collision

[0152] The collision rate has been reduced by nearly 40%. Using a 3D-reconstructed greenhouse simulation environment and deep learning models for object segmentation, the robot can accurately locate fruit and obstacles. During path planning, the collision detection module provides real-time feedback to avoid occlusion and collisions.

[0153] 4. Optimize resource utilization

[0154] Reinforcement learning model training in a virtual greenhouse avoids repeated debugging in a real environment, reducing hardware wear and energy consumption. Furthermore, the digital twin model simulates plant growth trends and predicts the optimal harvest time, reducing fruit waste caused by picking too early or too late.

[0155] 5. Demonstrate superior performance

[0156] Compared to picking solutions based on 2D images, the digital twin model environment of this invention provides precise spatial information about the fruit, solving the problem of occlusion and complex spatial distribution that 2D decision-making cannot handle. Furthermore, the simulation environment supports the simulation of the dynamic growth trends of the fruit, enabling more precise picking timing for the robot.

[0157] 6. Comprehensive Effect

[0158] Improved efficiency: Picking efficiency is increased by more than 30%, reducing picking time and labor costs.

[0159] Motion optimization: The robot arm's moving distance is shortened by more than 20%, reducing energy consumption and equipment wear.

[0160] Enhanced reliability: The collision rate is reduced by nearly 40%, effectively protecting plants and fruits and improving the reliability of the robot system.

[0161] Resource saving: Through virtual training and precise picking, debugging costs and fruit waste are saved.

[0162] In general, the present invention not only realizes the efficiency, precision and intelligence of the harvesting robot, but also demonstrates strong advantages in resource conservation and environmental adaptability, promoting the development of agricultural modernization and intelligent harvesting technology.

[0163] The above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention. The scope of patent protection of the present invention should be defined by the claims.

Claims

1. A greenhouse fruit picking method based on a digital twin model, characterized in that: include: Collect basic greenhouse data, as well as attribute data and growth data of crops in the greenhouse, and build a digital twin model of the crop based on the basic data, attribute data, and growth data. The basic data includes the spatial dimensions, planting layout, and environmental data of the greenhouse. The attribute data includes the external dimensions, color, texture, and picking requirements of the crop's fruits at various growth stages. The growth data includes the current external dimensions, color, texture, and current plant dimensions of the crop. Based on the digital twin model, the current maturity information of the fruit is obtained, and a current picking plan is generated based on the current maturity information; Driving the picking robot to perform picking operations according to the current picking plan; Based on the digital twin model, the ripening trend information of the fruit is predicted, and a picking plan for future picking operations is generated based on the ripening trend information; Future harvest preparations are carried out based on the harvest plan.

2. The greenhouse fruit picking method according to claim 1, wherein: The steps to generate a picking plan include: Obtaining historical image data of fruits at various growth stages to generate a maturity training set, training a reinforcement learning network, and obtaining a maturity judgment model; the maturity judgment model generates the current maturity information, and based on the current maturity information, all fruits are identified as mature fruits or unripe fruits; A picking training set is generated based on the historical picking records of the crop, and a reinforcement learning network is trained to obtain a picking plan generation model; the current picking plan is generated through the picking plan generation model.

3. The greenhouse fruit picking method according to claim 2, wherein: The steps to generate a picking model include: Based on the digital twin model, all the fruits are 3D modeled to generate a picking model, which includes the plant dimensions of the crop where the fruit is located and the spatial position of the fruit; based on the picking plan generation model and the picking model, the current picking plan is generated.

4. The greenhouse fruit picking method according to claim 2, wherein: The steps of generating a harvesting plan include: training a reinforcement learning network with the maturity training set to obtain a maturity trend prediction model; The steps to prepare for future harvesting include: Based on the digital twin model and the ripening trend prediction model, the ripening period of the immature fruit is predicted, and based on the ripening period, a picking plan for the immature fruit is generated; Before the fruit is picked, the picking plan is adjusted according to the fruit picking demand.

5. A greenhouse fruit picking device based on a digital twin model, characterized in that: include: A model building module, used to build a digital twin model of the crop; include , collecting basic data of the greenhouse, as well as attribute data and growth data of the crops in the greenhouse, and constructing the digital twin model based on the basic data, the attribute data and the growth data; the basic data includes the internal space dimensions and planting layout of the greenhouse, the attribute data includes the external dimensions, color texture and picking requirements of the fruits of the crops at various growth stages, and the growth data includes the current external dimensions, color texture of the fruits, and the current external dimensions of the crops; A picking module is used to obtain the current maturity information of the fruit based on the digital twin model and generate a current picking plan based on the current maturity information; Driving the picking robot to perform picking operations according to the current picking plan; A prediction module is used to predict the ripening trend information of the fruit based on the digital twin model, and generate a picking plan for future picking operations based on the ripening trend information; Future harvest preparations are carried out based on the harvest plan.

6. The greenhouse fruit picking device according to claim 5, characterized in that: The picking module includes: A maturity model training module is used to obtain historical image data of fruits at various growth stages to generate a maturity training set, train a reinforcement learning network, and obtain a maturity judgment model; the maturity judgment model generates the current maturity information, and based on the current maturity information, all fruits are identified as mature fruits or unripe fruits; The picking model training module is used to generate a picking training set based on the historical picking records of the crop, train the reinforcement learning network, and obtain a picking plan generation model; and generate the current picking plan through the picking plan generation model.

7. The greenhouse fruit picking device according to claim 6, characterized in that: The picking model training module includes: The 3D modeling module is used to perform 3D modeling of all the fruits based on the digital twin model and generate a picking model. The picking model includes the plant dimensions of the crop where the fruit is located and the spatial position of the fruit; based on the picking plan generation model and the picking model, the current picking plan is generated.

8. The greenhouse fruit picking device according to claim 6, characterized in that: The prediction module includes: The trend model training module trains the reinforcement learning network with the maturity training set to obtain a mature trend prediction model; a plan generation module, configured to predict the maturity period of the immature fruit based on the digital twin model and the maturity trend prediction model, and generate a plan for picking the immature fruit based on the maturity period; The plan adjustment module is used to adjust the picking plan according to the fruit picking demand before the fruit is picked.

9. An electronic device comprising a greenhouse fruit picking device based on a digital twin model as described in any one of claims 5 to 8.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer-executable instructions are executed, the greenhouse fruit picking method based on the digital twin model as described in any one of claims 1 to 4 is implemented.

Images