Method for picking fruits in non-structured scene based on whole-process bionic fruit data monitoring

Abstract

The application discloses a kind of non-structure scene fruit picking methods based on whole-process bionic fruit data monitoring.The method comprises the following steps: constructing a multi-layer perception structure bionic fruit, the outer layer structure is used for force perception and simulating the color of real fruit;then the end effector of picking robot is identified, grabbed and picked bionic fruit showing the mature color of real fruit, while bionic fruit real-time acquisition grabbing data, so as to monitor the whole process of picking bionic fruit in real time.The application method proposes robot-fruit-environment interaction application mode, and machine-fruit closed-loop interaction mode, which can realize dynamic simulation and multi-modal data acquisition of whole process of fruit picking in non-structure scene, accurately characterize the damage mechanism and mechanical properties of picking process, break through the bottleneck of traditional picking process data acquisition, and provide a new path for agricultural data acquisition.

Description

TECHNICAL FIELD

[0001] The present application relates to a fruit picking method, in particular to a non-structured scene fruit picking method based on whole-process bionic fruit data monitoring. BACKGROUND

[0002] As a high-value agricultural product, fruit occupies an important position in market size, economic value, and supply chain demand. Fruit picking, as an important link in the production process of fruit, has always been a hot topic in the field of agricultural engineering.

[0003] In recent years, with the development and maturity of artificial intelligence technology, intelligent robot technology has made significant progress in agricultural scenarios, and the emerging agricultural embodied intelligence technology has gradually entered the public eye. However, during the application and promotion of these technologies, the lack of real-world data has always been an important factor restricting their development.

[0004] In the development of fruit picking technology, researchers strive to find an efficient, low-damage, and high-success rate automated fruit picking method. However, these methods are highly dependent on the acquisition of fruit information such as location, ripeness, occlusion condition, size, shape, orientation, etc. These information collectively determine the accuracy, efficiency, and effectiveness of the picking operation.

[0005] There are two main data acquisition methods for existing fruit picking technology: one is the traditional picking method based on real fruit picking, and the other is the picking method based on static simulation model. Both methods have the following problems: traditional real fruit picking: data is difficult to obtain, information is difficult to quantify, and the process cannot be repeated. Static simulation model picking: simulation effect is limited, simulation link is single, and data cannot be collected. These problems directly or indirectly lead to high missed picking rate, low accuracy, and low effectiveness in the fruit picking process, making it difficult to balance the comprehensive needs of fruit damage, picking efficiency, and complete picking in the application and promotion of technology.

[0006] In summary, in the complex, dynamic, and unstructured orchard environment, it is difficult to obtain the key multi-dimensional data and information required for the research, development, and application of fruit picking technology in real time, accurately, and at low cost. To solve the existing picking efficiency, accuracy, and effectiveness problems. SUMMARY

[0007] In order to solve the problems in the background art, the present application provides a non-structured scene fruit picking method based on whole-process bionic fruit data monitoring. The method of the present application can solve the problems of low picking efficiency, insufficient precision, high damage rate and poor effectiveness caused by the difficulty in obtaining key data in the process of research and development and application promotion of artificial intelligence and robot technology, and can realize non-structured scene fruit picking driven by bionic fruit whole-process, digitization and multi-functional data monitoring. The method of the present application is driven by a set of bionic fruit capable of realizing whole-process, digitization and multi-functional data monitoring. The bionic fruit can collect, store, record and wirelessly transmit multi-dimensional information data such as fruit color, position, motion and damage required in the picking process. The present application can simulate and simulate the whole process of identification, picking and collection involved in fruit picking, effectively meet the simulation and data acquisition needs in the whole cycle of scientific research, application development and product promotion, and provide key technical support for solving the data acquisition bottleneck problem in agricultural scenes, especially in the fruit picking process, and improving the automation and intelligent level in this field.

[0008] The technical solution adopted by the present application is:

[0009] The non-structured scene fruit picking method based on whole-process bionic fruit data monitoring of the present application comprises:

[0010] Step 1) Construct a data monitoring bionic fruit with a multi-layer perception structure. The outer structure of the data monitoring bionic fruit is used for force perception and simulation of the color of real fruit.

[0011] Step 2) Identify and pick the data monitoring bionic fruit showing the mature color of real fruit through the end effector of the picking robot, and at the same time, the data monitoring bionic fruit obtains real-time grabbing data to represent the physical response characteristics of the data monitoring bionic fruit in the whole picking process, so as to monitor the whole picking process of the data monitoring bionic fruit in real time, obtain key multi-dimensional data and information in the fruit picking process, and improve the picking efficiency and precision, and realize non-destructive picking.

[0012] The step 1) includes a plurality of epidermal sensing micro-unit cells, an outer structure layer net unit, a force sensing framework unit, a cellular structure pulp unit and a signal energy supply structure unit arranged from outside to inside to simulate the upper epidermis layer, the outer structure layer, the lower epidermis layer, the pulp layer and the inner structure layer of the real fruit, i.e. the outer pericarp, the mesocarp, the endocarp, the fruit core, and balance the needs of authenticity and functionality; the outer structure of the data monitoring bionic fruit includes each epidermal sensing micro-unit cell for contact sensing and simulating the color of the real fruit, the outer structure layer net unit for supporting each epidermal sensing micro-unit cell, and the force sensing framework unit for sensing the grasping pressure; the data monitoring bionic fruit also includes a shear sensing unit to simulate the fruit stem, the top end of the signal energy supply structure unit is located at the top center of the data monitoring bionic fruit where the shear sensing unit is installed outside each epidermal sensing micro-unit cell, and the bottom end of the shear sensing unit is connected to the top end of the signal energy supply structure unit; each epidermal sensing micro-unit cell, the force sensing framework unit and the shear sensing unit are electrically connected to the signal energy supply structure unit. The data monitoring bionic fruit is used to simulate the color change, touch, pressure, position and motion path involved in the identification, grasping and collection stages in the fruit picking process, and real-time data acquisition.

[0013] The signal energy supply structure unit includes a communication module, an energy supply management module, a microcontroller unit (MCU) module, an inertial measurement unit (IMU) module, an inner structure layer framework and a wireless charging module. The inner structure layer framework is integrally formed by a hollow spherical main body framework and two central shaft channels to simulate the core of the fruit. The two central shaft channels are vertically arranged at the top center and the bottom center of the spherical main body framework, respectively. The top end of the upper central shaft channel is connected to the bottom end of the shear sensing unit. The outer peripheral surface of the spherical main body framework is covered with a second substrate layer. The communication module, the energy supply management module, the microcontroller unit (MCU) module, the inertial measurement unit (IMU) module and the wireless charging module are all installed inside the spherical main body framework and isolated by a plurality of insulating partition plates. The wireless charging module includes an infinite charging coil and a coil base. The coil base is installed inside the lower part of the inner structure layer framework. The infinite charging coil is wound on the coil base and electrically connected to the energy supply management module to supply energy to the internal modules through wireless charging. The communication module, the energy supply management module and the inertial measurement unit (IMU) module are electrically connected to the microcontroller unit (MCU) module through electrical connection lines. The microcontroller unit (MCU) module is electrically connected to each epidermal sensing micro-unit cell, the force sensing framework unit and the shear sensing unit through interlayer electrical connection lines in the two central shaft channels. The interlayer electrical connection lines are mainly used for data transmission, signal control and power supply between each level of the fruit and the signal energy supply structure unit, as well as mechanical and electrical connection with the shear sensing unit.

[0014] The shear sensing unit comprises two fruit stem segments, the first fruit stem segment is integrally formed with a ring-shaped hook and a fruit stem stem, the ring-shaped hook is located at the top end of the fruit stem stem to suspend the data monitoring bionic fruit whole at a fixed position, a first fruit stem cavity is opened in the bottom of the fruit stem stem and a permanent magnet is installed inside, a second fruit stem cavity is opened in the second fruit stem segment, a field coil, an iron core and a magnetic attraction meter are installed in the second fruit stem cavity, the field coil is wound on the iron core, the magnetic attraction meter is installed at one end of the iron core, and the field coil and the magnetic attraction meter are electrically connected to the micro control unit MCU module through the signal connection line and the interlayer electrical connection line; the field coil can be energized to exert different adsorption forces through the relationship between different fruit types and maturity and magnetic attraction, when the field coil is energized, the end of the permanent magnet and the other end of the iron core are adsorbed, at this time, the bottom end of the first fruit stem segment and the top end of the second fruit stem segment are connected and the two fruit stem cavities are communicated; when the data monitoring bionic fruit is picked, the separation of the permanent magnet and the iron core simulates the fruit stem cutting process, and the magnetic attraction meter obtains the magnetic attraction when the permanent magnet and the iron core are separated to feed back the shear force threshold at the moment of fruit stem cutting to provide a basis for the actual picking process.

[0015] The force sensation perception frame unit comprises a flexible frame, a plurality of pressure sensor arrays and a plurality of isolation layers, the flexible frame is a hollow spherical structure, the signal energy supply structure unit is located in the inner center of the flexible frame, the top surface and the bottom surface center of the flexible frame are both provided with a fruit stem interface and are respectively sleeved on the upper and lower two center shaft channels of the structural layer frame, the circumferences of the fruit stem interface and the center shaft channel are both provided with a plurality of electrical interfaces to be electrically connected through the electrical connection lines; each isolation layer is sealingly connected on the flexible frame as the surface of the flexible frame and does not cover the two fruit stem interfaces, each pressure sensor array is arranged on the outer surface of a respective isolation layer, and no pressure sensor array is arranged on the isolation layer close to the two fruit stem interfaces, each pressure sensor array comprises a plurality of uniformly spaced pressure sensors and is electrically connected to the micro control unit MCU module through the electrical interfaces and the electrical connection lines arranged in the isolation layer, and each pressure sensor array integrates the pressure data of the respective pressure sensors and transmits the pressure data to the micro control unit MCU module.

[0016] The cell structure pulp unit is composed of a plurality of layers of cell structure hydrogel arrays, and the cell structure pulp unit is filled between each isolation layer of the force sensation perception frame unit and the second substrate layer of the inner structural layer frame of the signal energy supply structure unit.

[0017] The outer structural layer mesh unit is a spherical structure composed of several triangles forming a mesh-like envelope. The flexible frame of the force sensing frame unit is internally connected to the outer structural layer mesh unit but not in contact with it. The entire frame of the outer structural layer mesh unit is made of flexible tubes, and the connection points of the triangular frames formed by the flexible tubes serve as self-positioning nodes. The flexible tubes are mainly used for the arrangement of control signal lines, data signal lines, and power supply lines. A microchip is installed on the inner side of each self-positioning node through a thin plastic sheet. Each microchip has a different position number and is electrically connected to the microcontroller unit (MCU) module. Each triangular frame has a skin sensing micro-element unit installed and corresponds to its own microchip and its position number. Each skin sensing micro-element unit is electrically connected to the MCU module. The top and bottom centers of the outer structural layer mesh unit form stalk connection ports through the ends of several flexible tubes. The two stalk connection ports are respectively connected to the two central axis channels of the structural layer frame. The stalk connection ports are also used to converge the control signal lines, data signal lines, and power supply lines into the central axis channels to achieve electrical connection with the MCU module. The main function of the outer structural layer mesh unit is to provide mechanical support and electrical connection for the surface of the simulated fruit, so that the epidermal sensing micro-element unit can be effectively attached.

[0018] Each of the aforementioned epidermal sensing micro-element units includes several tactile sensing micro-elements, a first substrate layer, and an electrochromic layer. For each triangular frame composed of a flexible tube, the first substrate layer is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit. The electrochromic layer is attached to the outer side of the first substrate layer. Each electrochromic layer completely encloses the outer structural layer mesh unit and serves as the outer surface. The inner side of the first substrate layer is uniformly spaced with several tactile sensing micro-elements. Each tactile sensing micro-element faces the pressure sensor array but does not contact it. Each tactile sensing micro-element and the electrochromic layer are electrically connected to a microcontroller unit (MCU) module. The MCU module applies voltage to the electrochromic layer to change color according to the different microchip positions.

[0019] In step 2), the harvesting process of the data-monitored bionic fruit is divided into an identification stage, a harvesting stage, and a collection stage. In the identification stage, the microcontroller unit (MCU) module applies voltage to each electrochromic layer based on the location number of the microchip on the self-positioning node to simulate the ripe color of real fruit. When the harvesting robot recognizes that the electrochromic layer displays the ripe color of real fruit, it controls the end effector to grasp the data-monitored bionic fruit and enter the harvesting stage. In the harvesting stage, when the end effector touches each tactile sensing micro-element, and a piezoresistive change occurs in the tactile sensing micro-element, it determines that the end effector has made contact and thus achieves harvesting. Tactile sensing is achieved as the end effector continues to grasp the material until an array of pressure sensors generates pressure data for pressure sensing. Finally, the permanent magnet and iron core are separated, and the shear force at the moment of separation is obtained through a magnetic force gauge for shear sensing. Then, the process enters the collection phase. During the collection phase, the end effector grasps the harvested data-monitoring bionic fruit and moves it to a position directly above the fruit basket. Finally, the end effector is released to place the data-monitoring bionic fruit into the basket. The inertial measurement unit (IMU) module acquires real-time acceleration data of the data-monitoring bionic fruit as it moves and falls with the end effector for motion sensing.

[0020] Before the identification stage, the data monitoring bionic fruit is first suspended by a ring hook, and the excitation coil is energized by the microcontroller unit (MCU) module to apply a preset attraction force between the permanent magnet and the iron core.

[0021] The beneficial effects of this invention are:

[0022] 1) Achieving dynamic simulation and multimodal data acquisition of the entire fruit harvesting process in unstructured scenarios: The main components of this invention cover three stages: identification, harvesting, and collection. In the identification stage, electrochromic technology can be used to dynamically simulate fruit color changes. In the harvesting stage, tactile and pressure sensors can be integrated to simulate the clamping force state, and shearing force can be accurately simulated through magnetic force calculation, allowing for real-time capture of harvesting actions and positional changes. In the collection stage, the positional changes of the fruit can be continuously tracked.

[0023] 2) Precise characterization of damage mechanisms and mechanical properties during the harvesting process: This invention can effectively quantify and record key parameters such as touch, force, and pressure. By summarizing and analyzing multi-source data, and using the mapping relationship between fruit damage mechanisms and mechanical properties obtained from external reference instruments and the characteristics of sensor simulated signals as references, a visualized damage and force characterization map is generated, filling the gap in dynamic and in-situ data acquisition and characterization in this field.

[0024] 3) Breaking through the bottleneck of traditional harvesting process data acquisition: This invention overcomes the difficulty of data collection in real unstructured environments or static models, providing a new path for agricultural data acquisition. Simultaneously, it innovatively proposes a robot-fruit-environment interaction application mode with sensors externally placed in the acquisition system, as well as a new method for closed-loop interaction between the machine and fruit.

[0025] 4) Provides highly customizable fruit simulation: This invention can flexibly adjust the growth parameters of the fruit, such as color, firmness, and stem characteristics, according to research needs, and achieve customized design in terms of weight, size, and shape, so as to provide full-process data for the real harvesting process. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall structure of the data monitoring bionic fruit provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the physical levels of data monitoring for biomimetic fruits provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the epidermal sensing micro-element unit of a biomimetic fruit for data monitoring, provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of the outer structure layer mesh unit of the biomimetic fruit for data monitoring provided in an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of the force perception framework unit for data monitoring of bionic fruits provided in an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the cellular structure of the pulp layer of a biomimetic fruit for data monitoring, provided in an embodiment of the present invention.

[0032] Figure 7 This is a structural diagram of the signal power supply structure unit for data monitoring of a biomimetic fruit provided in an embodiment of the present invention;

[0033] Figure 8 A three-dimensional view of the signal power supply structure unit of the bionic fruit for data monitoring provided in this embodiment of the invention;

[0034] Figure 9 This is a cross-sectional view of the shear sensing unit of the biomimetic fruit for data monitoring provided in an embodiment of the present invention;

[0035] Figure 10 This is a system architecture diagram for monitoring biomimetic fruits according to an embodiment of the present invention;

[0036] Figure 11 This is a diagram illustrating the architecture of the harvesting process simulation steps provided in an embodiment of the present invention.

[0037] Figure 12 A flowchart of an unstructured fruit picking method based on biomimetic fruit data monitoring driven by an embodiment of the present invention;

[0038] Figure 13 Reference data diagrams of texture analyzer for the harvesting process provided in embodiments of the present invention;

[0039] Figure 14 A pressure data graph of a capacitive flexible sensor during the harvesting process provided in an embodiment of the present invention;

[0040] Figure 15 An acceleration simulation curve of the harvesting process provided in an embodiment of the present invention, wherein, Figure 15 (a) is a simulated acceleration curve when the robotic arm approaches the fruit. Figure 15 (b) is a simulated acceleration curve of the robotic arm placement motion. Figure 15 (c) is a simulated acceleration curve of the robotic arm collecting the basket.

[0041] In the diagram: 1. Epidermal sensory micro-element unit; 101. Tactile sensory micro-element; 102. First substrate layer; 103. Electrochromic layer; 2. Outer structure layer mesh unit; 201. Flexible tube; 202. Self-positioning node; 203. Fruit stem connection port; 3. Force sensory frame unit; 301. Flexible frame; 302. Pressure sensor array; 303. Isolation layer; 304. Fruit stem interface; 4. Cellular structure pulp unit; 5. Signal power supply structure unit; 501. Central axis channel; 502. Communication module; 503. Electrical connection line; 5 04. Power supply management module; 505. Microcontroller unit (MCU) module; 506. Inertial measurement unit (IMU) module; 507. Inner structural layer frame; 508. Insulating partition plate; 509. Wireless charging coil; 510. Coil base; 511. Interlayer electrical connection line; 6. First fruit stalk segment; 601. Ring hook; 602. Fruit stalk stem; 603. Permanent magnet; 604. First fruit stalk cavity; 7. Second fruit stalk segment; 701. Excitation coil; 702. Magnetic force meter; 703. Signal connection line; 704. Second fruit stalk cavity. Detailed Implementation

[0042] To facilitate understanding of the present invention, it will be further described in detail below with reference to the accompanying drawings and specific embodiments. Specific embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0043] The present invention provides a non-structured fruit harvesting method based on full-process biomimetic fruit data monitoring, as detailed below:

[0044] Step 1) Construct a data monitoring bionic fruit with a multi-layered sensing structure. The outer layer of the data monitoring bionic fruit is used for force sensing and simulating the color of real fruit. The data monitoring bionic fruit used in this embodiment of the invention for driving a fruit-picking method in unstructured scenarios mainly includes two parts: a simulated fruit stalk and a simulated fruit. The simulated fruit stalk is a curved hollow columnar structure made of polylactic acid (PLA) rigid plastic. The simulated fruit has a multi-layered structure, with an outer layer of flexible material and an inner layer of rigid material. For example... Figure 1 and Figure 2 As shown, the multi-layer sensing structure of the biomimetic fruit for data monitoring includes several epidermal sensing micro-units 1, outer structural layer mesh units 2, force sensing frame units 3, cellular pulp units 4, and signal power supply structure units 5 arranged sequentially from the outside to the inside. This simulates the upper epidermis, outer structural layer, lower epidermis, pulp layer, and inner structural layer of a real fruit, namely, the pericarp, mesocarp, endocarp, and pit, while balancing the needs of realism and functionality. In this embodiment, the main layers of the simulated fruit's multi-layer structure are: the upper epidermis with epidermal sensing micro-units 1 as the main functional structure, the outer structural layer with outer structural layer mesh units 2 as the main functional structure, the lower epidermis with force sensing frame units 3 as the main functional structure, the pulp layer with cellular pulp units 4 as the main functional structure, and the signal and power supply layer with signal power supply structure units 5 as the main functional structure. The upper epidermis layer is attached above the outer structural layer, with a gap of about 1-2 mm between the upper and lower epidermis layers to prevent the pressure sensor of the lower epidermis layer from being triggered. The pulp layer is attached to the inner surface of the lower epidermis layer and extends to the outer side of the inner structural layer. The inner structural layer and the lower epidermis layer together provide support for the pulp layer. The signal and power supply layer is distributed in the inner layer of the inner structural layer, located in the core of the simulated fruit. Specifically, in this embodiment, there is a gap of about 1 mm between the force sensing frame unit 3 and the outer structural layer mesh unit 2 to prevent false triggering caused by the self-compression between the tactile sensing micro-element 101 and the pressure sensor array 302. The outer structural layer mesh unit 2 envelops the outside of the force sensing frame unit 3, the epidermal sensing micro-element unit 1 is attached to its surface, and the cellular pulp unit 4 is attached to the inner side of the isolation layer 303 of the force sensing frame unit 3 and extends to the outer side of the inner structural layer frame 507 of the signal power supply structure unit 5. The figure is only for schematic representation.

[0045] The outer structure of the data monitoring bionic fruit includes individual epidermal sensing micro-units 1 for contact sensing and simulating the color of real fruit, an outer structural mesh unit 2 for supporting each epidermal sensing micro-unit 1, and a force sensing frame unit 3 for sensing grasping pressure. The data monitoring bionic fruit also includes a shearing sensing unit to simulate the stem of a real fruit. The top of the signal power supply structure unit 5 is located at the top center of the shearing sensing unit, outside each epidermal sensing micro-unit 1, and the bottom of the shearing sensing unit is connected to the top of the signal power supply structure unit 5. Each epidermal sensing micro-unit 1, force sensing frame unit 3, and shearing sensing unit is electrically connected to the signal power supply structure unit 5. The data monitoring bionic fruit is used to simulate and model the fruit color changes, touch, pressure, position, and movement path involved in the identification, grasping, and collection stages during fruit picking, and to acquire corresponding data in real time.

[0046] like Figure 7 and Figure 8As shown, the signal power supply structure unit 5 includes a communication module 502, a power management module 504, a microcontroller unit (MCU) module 505, an inertial measurement unit (IMU) module 506, an inner structural frame 507, and a wireless charging module. The inner structural frame 507 is integrally formed from a hollow spherical main frame and two central axis channels 501 to simulate the core of a fruit. The two central axis channels 501 are vertically arranged at the top center and bottom center of the spherical main frame, respectively. The two central axis channels 501 are hollow tubular structures made of hard plastic or hard rubber. The top of the upper central axis channel 501 is connected to the shear sensing unit. At the bottom, the outer periphery of the spherical main frame is covered with a second substrate layer, specifically made of transparent latex material. The communication module 502, power management module 504, microcontroller unit (MCU) module 505, inertial measurement unit (IMU) module 506, and wireless charging module are all installed inside the spherical main frame and isolated by several insulating partitions 508. Due to the overall weight of the fruit, the inner structural frame 507 and the insulating partitions 508 need to possess a certain degree of rigidity without being too heavy. Therefore, the inner structural frame 507 is specifically made of a 5-10 mm thick hard low-density alloy such as magnesium-lithium alloy or carbon fiber material, and the insulating partitions... The partition 508 is specifically constructed of a hard, low-density alloy material such as magnesium alloy and an insulating film. The insulating partition 508 has a central cutout to allow for the passage of electrical connection wires 503. The wireless charging module includes a wireless charging coil 509 and a coil base 510. The coil base 510 is installed in the lower part of the inner structural frame 507. The wireless charging coil 509 is wound on the coil base 510 and electrically connected to the power management module 504 to supply power to the internal modules via wireless charging. The communication module 502, the power management module 504, and the inertial measurement unit (IMU) module 506 are all electrically connected to the microcontroller unit (MCU) module 505 via electrical connection wires 503. The communication module 502 can use a WIFI module and a Bluetooth module to wirelessly transmit the data of the microcontroller module 505 to an external computer device. The power management module 504 can use a battery protection board for power supply. The microcontroller module 505 is electrically connected to each epidermal sensing micro-element unit 1, force sensing frame unit 3 and shear sensing unit through interlayer electrical connection lines 511 located in the two central axis channels 501. The interlayer electrical connection lines 511 are mainly used for data transmission, signal control and power supply between the layers of the simulated fruit and the signal power supply structure unit 5, as well as mechanical and electrical connection with the shear sensing unit.

[0047] In this embodiment, the inner structural layer frame 507 and the insulating partition plate 508 are made of hard low-density alloy magnesium-lithium alloy. Alternatively, carbon fiber can also be used as a component material. The surface of the insulating partition plate 508 is wrapped with an insulating film to prevent the alloy or carbon fiber material from directly contacting the components and thus conducting electricity. The entire inner structural layer is divided by several insulating partition plates 508. In this embodiment, the insulating partition plates 508 divide the inner structural layer frame 507 into four layers, each used to install different circuit modules. The partition plates have a central cutout for electrical connection wires. The insulating partition plates 508 are circular in shape, and the arc-shaped ends of the inner structural layer frame 507 are connected to the central shaft channel 501. The coil base 510 is made of polypropylene plastic and is mechanically connected to the bottom of the lowest insulating partition plate 508. It also has a hollow structure in the middle. The wireless charging coil 509 spirals downwards from the connection point between the coil base 510 and the partition plate 508, tightly wound around the outer surface of the coil base 510. It is directly connected to the power management module 504 via an electrical connection line 503 through a hollow structure in the center of the coil base 510. The coil diameter of the wireless charging coil 509 is approximately 44 mm. The power management module 504 is powered via the electrical connection line 503 and is mounted on the upper surface of the bottom insulating partition plate 508. Specifically, the electrical connection line 503 connects adjacent layers through a central hollow structure in the insulating partition plate 608, facilitating communication between signals and modules in the power supply layer. In this embodiment, the charging and discharging management module uses an 18650 lithium battery charging and discharging integrated module. The microcontroller unit (MCU) module 505 uses a controller circuit composed of an STM32F407 chip and is mounted on the upper surface of the middle insulating partition plate 508. It features high performance and low power consumption, enabling it to effectively perform tasks such as signal control and data acquisition. The communication module 502 uses both Wi-Fi and Bluetooth modules, and is also mounted on the upper surface of the middle insulating partition plate 508, but is symmetrically distributed with the microcontroller unit (MCU) module 505. The Wi-Fi module is mainly used for data transmission, while the Bluetooth module is mainly used for communication between this device and other devices. The inertial measurement unit (IMU) module 506 uses an MMA8452Q and is mounted on the upper surface of the top insulating partition plate 508. Its small size and low power consumption make it suitable for miniaturized designs.

[0048] like Figure 9As shown, the shear sensing unit includes two curved hollow columnar fruit stalk segments 6 and 7, used to simulate the fruit stalk of a real fruit and to simulate the picking force during fruit harvesting and collect shear force data. Specifically, it can be made of polylactic acid (PLA). The first fruit stalk segment 6 is made of a plastic material of either acrylonitrile (Acid) or ABS (acrylonitrile-butadiene-styrene) terpolymer. It is integrally formed from a ring hook 601 and a fruit stalk stem 602. The ring hook 601 is located at the top of the fruit stalk stem 602 to suspend the entire data monitoring biomimetic fruit in a fixed position. A first fruit stalk cavity 604 is formed inside the bottom of the fruit stalk stem 602, and a permanent magnet 603 is installed inside. The permanent magnet 603 can be made of AlNiCo alloy. A second fruit stalk cavity 704 is formed inside the second fruit stalk segment 704. An excitation coil 701, a cylindrical iron core, and a magnetic force meter 702 are installed inside the second fruit stalk cavity 704. The excitation coil 701 is wound on the iron core, and the magnetic force meter 702 is installed at one end of the iron core. The excitation coil 701 and the magnetic force meter 702 are electrically connected to the microcontroller unit (MCU) module 505 via the signal connection line 703 and the interlayer electrical connection line 511. The excitation coil 701 can be energized to apply different adsorption forces based on the relationship between different fruit types, maturity levels, and magnetic attraction. When the excitation coil 701 is energized, the end of the permanent magnet 603 is attracted to the other end of the iron core. At this time, the bottom end of the first fruit stalk segment 6 and the top end of the second fruit stalk segment 7 are connected, and the two fruit stalk cavities 604 and 704 are connected. Data monitoring simulates the fruit stalk cutting process by separating the permanent magnet 603 and the iron core during harvesting. The magnetic force meter 702 obtains the magnetic force feedback at the moment of fruit stalk cutting to provide a basis for the actual harvesting process. In this embodiment, the curved hollow columnar structure simulating the fruit stalk is mainly divided into two parts: the first fruit stalk segment 6 and the second fruit stalk segment 7. The first fruit stalk segment 6 comprises a ring hook 601, a fruit stalk stem 602, and a first fruit stalk segment cavity 604. The second fruit stalk segment 7 is entirely hollow, and its second fruit stalk segment cavity 704 is made of PLA plastic using 3D printing. The permanent magnet 603 of the first fruit stalk segment 6 can be made of AlNiCo alloy. The excitation coil 701 of the second fruit stalk segment 7 is mainly used to generate magnetic forces of different magnitudes. The magnetic force meter 702 is mainly used to detect changes in shearing force during shearing or grasping and to collect data. The signal connection line 703 is used to transmit control signals, data signals, and power supply. The permanent magnet 603 of the first fruit stalk segment 6, along with the excitation coil 701, magnetic force meter 702, and signal connection line 703 of the lower fruit stalk segment, constitute a fruit stalk cutting sensor for feedback of shearing force.

[0049] like Figure 5As shown, the force sensing frame unit 3 includes a flexible frame 301, several pressure sensor arrays 302, and several isolation layers 303. The flexible frame 301 is a hollow spherical structure made of solid rubber columns or solid flexible columnar material. The hollow structure is similar to the distribution of latitude and longitude lines. The signal power supply structure unit 5 is located at the internal center of the flexible frame 301. The top and bottom surfaces of the flexible frame 301 are each provided with a fruit stem interface 304, which is respectively fitted onto the upper and lower central axis channels 501 of the structural layer frame 507. The fruit stem interface 304 and the central axis channels 501 are provided with several isolation layers 303 around their circumference. The electrical interface is connected via an electrical connection line; each isolation layer 303 is sealed and connected to the flexible frame 301 as the surface of the flexible frame 301 and does not cover the two fruit stem interfaces 304. The isolation layer 303 is made of a flexible material such as rubber with a certain thickness. Each pressure sensor array 302 is arranged on the outer surface of its own isolation layer 303. The isolation layer 303 near the two fruit stem interfaces 304 does not have a pressure sensor array. Each pressure sensor array 302 includes several pressure sensors arranged at even intervals and is electrically connected to the microcontroller unit (MCU) module 505 through an electrical interface and electrical connection line set in the isolation layer 303. Each pressure sensor array 302 integrates the pressure data of its own pressure sensors and transmits it to the microcontroller unit (MCU) module 505. The pressure sensors can be strain gauge pressure sensors or piezoresistive pressure sensors. The flexible frame 301 mainly provides mechanical support for the overall lower epidermis and can simulate the pressure feedback of a fruit with a certain hardness. The pressure sensor array 302 is used to collect relevant data on the pressure location and pressure intensity. It is mainly attached tightly to the isolation layer through a certain physical and chemical process. The pressure sensor array 302 is distributed in an array on the isolation layer 303, except for the portion near the fruit stem interface 304. The isolation layer 303, besides being used to attach the pressure sensor array 302, also serves to embed control signal lines, data signal lines, and power supply lines, converging them to the upper and lower fruit stem interfaces 304 to achieve electrical connection with the microcontroller unit (MCU) module 505, completely enveloping the entire lower epidermis of the fruit. The fruit stem interface 304 is also used to install and fix the central axis channel 501 to secure the entire simulated fruit. In this embodiment, the force sensing frame unit 3 is a spherical frame with a diameter of 95mm. The main supporting structure of the frame is a flexible frame 301, whose main component is a solid rubber column, providing mechanical support for the overall structure. Using this frame structure as a skeleton, the force sensing frame unit 3 also has isolation layers 303 distributed on it. The isolation layer 303 is mainly composed of rubber material. The part of its upper surface that contacts the pressure sensor array 302 has electrical interfaces that can be connected to the pressure sensor. Inside it, there are control signal lines, data signal lines and power supply lines, which are arranged in a certain pattern.With the horizontal central axis as the dividing line, electrical connection lines above the axis converge at the upper fruit stem interface 304, while those below the axis converge at the lower fruit stem interface 304. The fruit stem interface 304 is a ring-shaped hollow structure, primarily composed of rigid plastic or carbon fiber. Electrical interfaces are distributed around its interior, allowing all electrical connection lines of the force sensing frame unit 3 to be connected to the central axis channel 501, enabling data interaction and energy supply to the lower skin layer and the signal and power supply layer. In addition, the fruit stem interface 304 is also used to install and fix the simulated fruit stem segments 6 and 7 and the central axis channel 501.

[0050] like Figure 6 As shown, the cellular pulp unit 4 is composed of several layers of cellular hydrogel arrays. Each layer of cellular hydrogel array has several rows and columns of cellular hydrogel. The cellular pulp unit 4 fills the space between the isolation layers 303 of the force perception frame unit 3 and the second substrate layer of the inner structural frame 507 of the signal power supply structure unit 5. The cellular pulp unit 4 is tightly attached to the inner side of the isolation layer 303 through a certain physicochemical process and is distributed horizontally and extends to the second substrate layer to simulate the texture of pulp. The hydrogel can be a high water-retention natural polymer hydrogel or a synthetic gel. The array formed can be encapsulated by polydimethylsiloxane (PDMS) or thermoplastic polyurethane (TPU). In this embodiment, the cellular pulp unit 4 extends horizontally from the inner surface of the isolation layer 303 to the outer side of the inner structural layer frame 507. The two structures provide mechanical support for it. At the same time, the outermost and innermost layers of the cellular pulp unit 4 are respectively adhered to the inner side of the isolation layer 303 and the outer side of the inner structural layer frame 507.

[0051] like Figure 4As shown, the outer structural layer mesh unit 2 has a spherical structure and is a mesh-like envelope structure composed of several triangles. The flexible frame 301 of the force sensing frame unit 3 is connected to the interior of the outer structural layer mesh unit 2 but does not contact it. The entire frame of the outer structural layer mesh unit 2 is made of flexible tubes 201, which are made of colloidal material. The connection points of the triangular frames formed by the flexible tubes 201 serve as self-positioning nodes 202. The flexible tubes 201 are mainly used for the arrangement of control signal lines, data signal lines, and power supply lines. A microchip is installed on the inner side of each self-positioning node 202 through a thin plastic sheet. The microchip uses a decoder chip and circuit. The circuit can select each skin sensing micro-element unit 1 by address gating. Each microchip has a different position number and is electrically connected. The microcontroller unit (MCU) module 505 accurately reads and controls the target epidermal sensing micro-element 1 through these address codes. Each triangular frame houses one epidermal sensing micro-element 1, which corresponds to a microchip and its location number. Each epidermal sensing micro-element 1 is electrically connected to the MCU module 505. The top and bottom centers of the outer structural layer mesh unit 2 form fruit stem connection ports 203 through the ends of several flexible tubes 201. The two fruit stem connection ports 203 are respectively connected to the two central axis channels 501 of the structural layer frame 507. The fruit stem connection ports 203 are also used to converge control signal lines, data signal lines, and power supply lines into the central axis channels 501 to achieve electrical connection with the MCU module 505. The main function of the outer structural layer mesh unit 2 is to provide mechanical support and electrical connection for the simulated fruit surface, enabling the epidermal sensing micro-element 1 to fit effectively. The outer structural layer mesh unit 2 is distributed in an envelope shape on the outside of the simulated fruit. In this embodiment, the outer structural layer mesh unit 2 is formed by connecting five vertically parallel 12-sided polygons. All external connection points are distributed on a circle with a diameter of approximately 100 mm. Self-positioning nodes 202 are distributed at all external connection points. A micro decoder chip, its supporting circuit, and a thin plastic sheet are provided at the self-positioning node 202. A micro interface circuit is also distributed at the connection between the self-positioning node 202 and the epidermal sensing micro-element unit 1 for electrical circuit connection of the epidermal sensing micro-element unit 1. Each epidermal sensing micro-element unit 1 corresponds to one self-positioning node 202.

[0052] like Figure 3As shown, each epidermal sensing micro-element unit 1 includes several tactile sensing micro-elements 101, a first substrate layer 102, and an electrochromic layer 103. For each triangular frame composed of a flexible tube 201, the first substrate layer 102 is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit 2. The electrochromic layer 103 is attached to the outer side of the first substrate layer 102. Each electrochromic layer 103 completely covers the outer structural layer mesh unit 2 and serves as the outer surface. The inner side of the first substrate layer 102 is evenly spaced with several tactile sensing micro-elements 101. Each tactile sensing micro-element 101 faces the pressure sensor array 302 but does not contact each other. Each tactile sensing micro-element 101 and the electrochromic layer 103 are electrically connected to the microcontroller unit MCU module 505. The microcontroller unit MCU module 505 applies voltage to the electrochromic layer 103 to change color according to the different microchip positions. The first substrate layer 102 has the same dimensions as the triangular frame it is attached to, and its outer surface is a planar, convex curved, or triangular prism structure. The electrochromic layer 103 is completely attached to the outer surface of the first substrate layer 102. When it is a triangular prism structure, the electrochromic layer 103 is divided into three parts and completely attached to the three surfaces of the triangular prism. The first substrate layer 102 can be made of a rubber material or a flexible film material of a certain thickness. Flexible film materials include polyethylene terephthalate (PET) and polydimethylsiloxane (PDMS). The tactile sensing micro-element 101 can be made of capacitive or resistive piezoelectric material. The characteristic of this type of material is that when pressure is applied, the electrical values ​​such as capacitance and resistance change with the pressure. 03. Inorganic or organic electrochromic materials, such as tungsten trioxide (WO3), are used. These materials can reversibly change their optical properties under applied voltage. The tactile sensing micro-element 101 and the electrochromic layer 103 are integrated on the first substrate layer 102 through certain physical and chemical processes, and control signals, data signals, and power supply interfaces are reserved. This enables the simulation of the color of fruits at different ripeness levels in the fruit picking and identification process, helping to verify the actual function and recognition effect of external identification equipment and recognition algorithms. Furthermore, it can sense and collect data on accidental touches of the fruit by the end effector of the picking mechanism in the fruit picking and grasping approach process, helping to improve the accuracy of the fruit picking and grasping approach process.In specific implementation, the epidermal sensing micro-element units 1 are closely distributed on the surface of the simulated fruit at an angle of 30°-60° to the tangent of the curved surface in the vertical direction; the tactile sensing micro-element 101 has a radius of about 1.2 mm, a height of about 0.3 mm, and an interval of about 0.3 mm between adjacent micro-elements. It integrates capacitive piezoelectric material inside and integrates it into the smooth lower surface of the first substrate layer 102 through physical and chemical processes. The upper surface of the first substrate layer 102 is a triangular pyramid structure with a certain curvature made of rubber material, and its lower surface is a curved surface with a thickness of about 3-5 mm. Electrochromic color-changing material 103 is integrated on the curved surface. The micro-element is mainly made of tungsten trioxide (WO3) with a thickness of 1-3 mm and is distributed on the surface of the triangular pyramid structure. The first substrate layer 102 also integrates electrical connection lines and reserves interfaces.

[0053] like Figure 10 As shown, the biomimetic fruit system architecture of the present invention mainly includes seven parts: a fruit stalk, an upper epidermis, an outer structural layer, a lower epidermis, a pulp layer, an inner structural layer, and a signal and power supply layer. Key components include a shear force sensor in the fruit stalk, a tactile sensor micro-element 101 and an electrochromic material 103 in the upper epidermis, a frame composed of a flexible tube 201 in the outer structural layer, a flexible frame 301, a pressure sensor array 302, and an insulating layer 303 in the lower epidermis, cellular pulp units 4 in the pulp layer, an insulating partition plate 508 and an inner structural layer frame 507 in the inner structural layer, and a communication module 502, a microcontroller unit (MCU) module 505, an inertial measurement unit (IMU) module 506, and an infinite charging coil 509 in the signal and power supply layer.

[0054] Step 2) The end effector of the harvesting robot identifies, grasps, and harvests data-monitored bionic fruits that exhibit the color of ripe fruit. Simultaneously, the data-monitored bionic fruit acquires grasping data in real time to characterize its physical response characteristics throughout the harvesting process. This allows for real-time monitoring of the entire harvesting process, obtaining key multi-dimensional data and information to improve harvesting efficiency and accuracy, while achieving non-destructive harvesting. The harvesting process of the data-monitored bionic fruit is divided into an identification stage, a harvesting stage, and a collection stage. Before the identification stage, the data-monitored bionic fruit is first suspended by a ring hook 601, and the excitation coil 701 is energized by the microcontroller unit (MCU) module 505 to apply a preset attraction force between the permanent magnet 603 and the iron core.During the recognition phase, the microcontroller unit (MCU) module 505 applies voltage to each electrochromic layer 103 according to the location number of the microchip on the self-positioning node 202 to simulate the ripe color of real fruit. When the picking robot recognizes that the electrochromic layer 103 presents the ripe color of real fruit, it controls the end effector to grasp the data and monitor the bionic fruit to enter the picking phase. The recognition phase includes a fruit setting stage and a fruit recognition stage. The fruit setting stage specifically involves modifying the operating parameters of the bionic fruit wirelessly or via wired connection through host computer software or wireless programming software, such as setting the cutting parameters according to ripeness and fruit type. The force used ensures the fruit is picked just right without causing severe damage. Selecting the color of the fruit at the target ripeness level prepares the way for the recognition process. This allows for the setting of parameters for the biomimetic fruit; structural settings such as size and weight can be customized before the experiment begins. The fruit recognition process involves the harvesting robot system controlling the visual recognition module, typically an RGBD binocular camera or a monocular image sensor. This module approaches and aligns with the data monitoring biomimetic fruit. The upper epidermis of the fruit will display the set ripeness color. The visual recognition module may be equipped with a target detection Yolov8 model. The image processing unit acquires and analyzes visual images to obtain information such as maturity status, location information, harvestable status (judged based on maturity status), harvest success probability (based on a combination of maturity status and location information), and the biomimetic fruit distribution and pose. Then, it transmits the necessary parameters for the harvesting stage to the robot system control terminal, facilitating the robot system to execute the next operation. During the harvesting stage, when the end effector touches each tactile sensing micro-element 101, and a change in pressure resistance occurs in the tactile sensing micro-element 101, it is determined that the end effector has made contact and achieved tactile sensing. The end effector continues to grasp until the pressure sensor array 302 produces... Pressure data is generated to achieve pressure sensing, ultimately separating the permanent magnet 603 from the iron core. The shear force at the moment of separation is obtained through a magnetic force gauge 702 to achieve shear sensing. The harvesting stage includes an approach phase and a picking phase. Specifically, in the approach phase, the harvesting robot system executes a harvesting approach action based on the parameters transmitted by the vision module. This approach action is a preparatory action for the picking phase. During the approach phase, the end effector of the harvesting robot system needs to move to the preparatory position for the picking phase. However, there is a probability that the end effector will collide with the fruit during the approach phase, causing damage. Therefore, touch sensing can effectively record and detect this situation. In the picking phase, the harvesting robot system controls the end effector to move to the target grasping position and issues a grasping command. The system's end effector then picks the biomimetic fruit. Because the robot system's end effector needs to provide shear force to twist the fruit stem for effective harvesting, pressure is inevitably generated on the fruit during the direct grasping method. Pressure sensing can effectively detect this situation and record relevant data.In addition, shear sensing is located at the lower end of the fruit stalk. Before the picking action, the upper and lower ends of the stalk are tightly attracted. The magnitude threshold of the magnetic force can be controlled by controlling the excitation coil 701. The magnetic force meter 702 can record the magnitude of the shear force at the moment of cutting. Then, the collection stage begins. In the collection stage, the end effector grabs the picked data-monitoring bionic fruit and moves it to the placement position directly above the fruit basket. Finally, the end effector is released to place the data-monitoring bionic fruit into the fruit basket. The inertial measurement unit (IMU) module 506 acquires the acceleration data of the data-monitoring bionic fruit in real time during its movement and fall with the end effector to achieve motion sensing. The collection stage includes the placement motion and the basket collection. The placement motion is specifically the process by which the picking robot system moves the bionic fruit from the picking position to the placement position. During this process, the robot's movement should be as smooth as possible. Acceleration may cause pressure on the fruit, leading to secondary damage; motion sensing during the placement phase can detect and record acceleration fluctuations and height changes during the placement phase; the collection and basket-loading phase mainly refers to the process by which the harvesting robot system places the bionic fruit from its placement position to the fruit storage device. During this process, if the height when placing the fruit in the basket is too high, it may cause secondary damage due to the fruit falling; motion sensing during the collection and basket-loading phase can detect and record acceleration fluctuations during the placement phase; fruit pressure data can be characterized and mapped using the fruit surface pressure data collected by the texture analyzer, and a reasonable pressure range can be determined based on the texture analyzer.

[0055] like Figure 11As shown, the bionic fruit of this invention is mainly applied to the three main stages of fruit harvesting: the identification stage, the harvesting stage, and the collection stage. Specifically, it can realize color simulation, touch perception, shearing perception, pressure perception, path perception, and fall perception. In specific implementation, the harvesting process is as follows: data monitoring of the bionic fruit's parameter configuration; the harvesting robot system begins the harvesting process; data monitoring of the bionic fruit's electrochromic layer 103 simulates the set ripe fruit color; the harvesting robot system approaches the identification position and enters the identification stage; the harvesting robot system collects image information and analyzes and judges the fruit's color ripeness; the harvesting robot system obtains the identification result and enters the approach stage to harvest the ripe bionic fruit; data monitoring of the bionic fruit's tactile sensing micro-element 101 detects whether it has been touched; the harvesting robot system approaches the harvesting position and performs the harvesting. The pressure sensor array 302 of the bionic fruit records pressure data; the magnetic force meter 702 of the bionic fruit monitors the shear force; the picking robot system completes the picking and enters the collection stage; the inertial measurement unit (IMU) module 506 of the bionic fruit monitors the acceleration changes; the inertial measurement unit (IMU) module 506 of the bionic fruit monitors the falling acceleration; the picking robot system completes the picking process; the grasping data includes data from the tactile sensing micro-element 101, the pressure sensor array 302, the magnetic force meter 702, and the inertial measurement unit (IMU) module 506.

[0056] like Figure 12 As shown, this embodiment specifically provides a method for harvesting fruit in unstructured scenarios driven by biomimetic fruit data monitoring. It mainly focuses on the aforementioned three main stages and six core steps, specifically including the following steps:

[0057] S01. Bionic Fruit Parameter Configuration:

[0058] Before the fruit harvesting process begins, the parameter configuration step for the bionic fruit involves modifying the operating parameters of the core controller microcontroller module 505 of the bionic fruit via a host computer software or wireless programming software, either wirelessly or via wired connection. This process selects and sets the fruit color for the target ripeness level, preparing for the recognition step and realizing the parameter setting of the bionic fruit.

[0059] S02, The harvesting robot system begins the harvesting process:

[0060] The external fruit-picking robot system initiates the fruit-picking process.

[0061] S03, Bionic Fruit Simulation: Sets the ripeness and color of the fruit.

[0062] In this stage, the biomimetic fruit mainly simulates color by controlling the electrochromic layer 103 on the upper surface of the epidermal sensing micro-element unit 1 array of the upper epidermal layer through the microcontroller unit MCU module 505 of the signal and power supply layer.

[0063] S04. The harvesting robot system approaches the identification location and enters the identification phase:

[0064] In this stage, the harvesting robot system controls the visual recognition module, which is usually an RGBD binocular camera equipped with a Yolov8 recognition model that approaches and aligns with the bionic fruit. The upper epidermis of the bionic fruit will then display the corresponding set color.

[0065] S05. The harvesting robot system collects and analyzes image information:

[0066] In this stage, the visual recognition module of the harvesting robot system collects and analyzes visual images. The specific analysis information includes: maturity status, location information, harvestable status, probability of successful harvesting, and the distribution and pose of the biomimetic fruit. After the analysis is completed, the result parameters required for the harvesting stage are transmitted to the robot system control terminal to prepare for the next operation.

[0067] S06. The harvesting robot system obtains the result parameters and enters the approach phase:

[0068] In this stage, the vision module of the harvesting robot system transmits the resulting parameters to execute the harvesting approach action. The harvesting approach action is a preparatory action for the harvesting stage. During the approach stage, the end effector of the harvesting robot system needs to move to the preparatory position for the harvesting stage. However, there is a probability that the end effector will collide with the fruit during the approach stage, causing damage to the fruit. For example... Figure 15 As shown, this invention simulates the accelerator curve changes at different stages of harvesting using a microcontroller unit (MCU) module 505 and an inertial measurement unit (IMU) module 506. In this step, simulation data from an embodiment of this invention can be used. Figure 15 The characteristics of the fruit contact acceleration signal curve in (a) are used as a reference to determine whether a collision has occurred. For example... Figure 15 As shown in (a), the acceleration changes when the robotic arm approaches the fruit are simulated. When the simulated fruit is slightly touched, the internal inertial measurement unit (IMU) module 506 will sense a small change in the acceleration of the X-axis and Y-axis. This change can be used to characterize the accidental touch of the approach link.

[0069] S07. Tactile micro-element detection of whether the biomimetic fruit is touched:

[0070] In this stage, the bionic fruit mainly achieves touch perception by collecting data from the tactile sensing micro-element 101 on the lower surface of the epidermal sensing micro-element unit 1 array in the upper epidermal layer through the micro-control unit MCU module 505 of the signal and power supply layer. This can effectively avoid fruit damage caused by the above situation.

[0071] S08. The harvesting robot system approaches the harvesting location and performs the harvesting:

[0072] In this stage, the harvesting robot system controls the end effector to move to the target grasping position and issues a grasping command. The end effector of the harvesting robot system then picks up the biomimetic fruit.

[0073] S09, The pressure sensor of the bionic fruit records the pressure data:

[0074] In this stage, the microcontroller unit (MCU) module 505 of the signal and power supply layer achieves pressure sensing by collecting pressure data from the pressure sensor array 302 on the force sensing frame unit 3 of the lower epidermis layer. During the picking process, to achieve effective harvesting, the robot system's end effector needs to provide shearing force to twist the fruit stem; therefore, pressure is inevitably generated on the fruit during direct fruit grasping. Pressure sensing can effectively detect this situation and record relevant data. In this stage, methods such as... Figure 14 The capacitive flexible sensor is used as the pressure sensing method, and reference data from the embodiments of the present invention are used. Figure 13 Using the mapping curve between damage mechanism and mechanical properties as a reference, a relationship between pressure sensor data and damage characterization is established, such as... Figure 13 As shown, in this invention, the surface pressure of fruits of different ripeness and types measured by a texture analyzer can be used as a reference to establish a mapping relationship between damage mechanisms and mechanical properties during the harvesting process, such as the pressure measured at the same depth when pressing kiwifruit, plums, and peaches. Figure 14 As shown, this invention uses a capacitive flexible sensor as an example. This sensor can collect capacitance data values ​​of fruits of different ripeness and types under pressure. By combining this value with... Figure 13 The obtained texture instrument values ​​can be used to obtain the mapping relationship between pressure data and electrical values ​​through certain mathematical operations and changes.

[0075] S10, the biomimetic fruit stalk cutting sensor provides feedback on shear force:

[0076] In this stage, the microcontroller unit (MCU) module 505 of the signal and power supply layer achieves shear sensing by collecting data from the fruit stalk cutting sensor. Before the picking action is completed, the upper and lower ends of the fruit stalk are tightly attracted. The MCU module 505 can control the magnitude threshold of the magnetic force by controlling the excitation coil 701. The lower magnetic force meter 702 can record the magnitude of the shear force at the moment of cutting.

[0077] S11. The harvesting robot system completes the harvesting and enters the collection stage:

[0078] The collection phase mainly covers the entire process of placing the simulated fruit into the storage device, including the placement movement stage and the collection and basketing stage. The placement movement stage refers to the process by which the picking robot system moves the bionic fruit from the picking position to the placement position. During this process, the robot's movement should be as smooth as possible; excessive acceleration may press on the fruit, causing secondary damage. The collection and basketing stage refers to the process by which the picking robot system places the bionic fruit from the placement position into the fruit storage device. During this process, if the height of the fruit placed in the basket is too high, it may cause secondary damage due to the fruit falling. In this stage, simulation data from this embodiment of the invention can be used. Figure 15 (b) and Figure 15 Using the acceleration signal curves of fruit placement and basket entry in (c) as a reference, we can determine whether there are sudden changes in the robotic arm's movement or fall damage. For example... Figure 15 As shown in (b), the acceleration changes during the placement motion of the robotic arm are simulated. When the simulated fruit is placed, a plateau-shaped acceleration change appears along the X-axis. This plateau-like change can be used to characterize the abrupt change in the robotic arm's motion during the placement process. Figure 15 As shown in (c), in order to simulate the acceleration change during the collection and basket entry process, when the simulation shows the fruit directly entering the basket from a certain height, a large Z-axis acceleration peak will appear at the moment the fruit touches the bottom of the basket. This Z-axis acceleration peak can be used to characterize the fruit collision during the collection and basket entry process.

[0079] S12. The accelerometer of the bionic fruit records the acceleration changes:

[0080] In this stage, the bionic fruit mainly uses the microcontroller unit (MCU) module 505 of the signal and power supply layer to control the inertial measurement unit (IMU) module 506 to collect acceleration change data of the placement path, thereby realizing the motion perception of the placement motion stage.

[0081] S13. The accelerometer of the bionic fruit records the intensity of the fall:

[0082] In this stage, the bionic fruit mainly uses the microcontroller unit (MCU) module 505 of the signal and power supply layer to control the inertial measurement unit (IMU) module 506 to collect acceleration change data of the placement path, thereby realizing the sensing of the falling intensity during the collection and basket insertion process.

[0083] S14. The harvesting robot system completes the harvesting process:

[0084] The external fruit-picking robot system has ended the fruit-picking process.

[0085] Specifically, during the aforementioned harvesting process, the microcontroller unit (MCU) module 505 of the signal and power supply layer controls the communication module 502 to send the collected data to the external data acquisition terminal in real time, thereby realizing data monitoring of the entire fruit harvesting process.

[0086] This embodiment uses real-time data monitoring of simulated fruits to drive fruit harvesting in unstructured scenarios, realizing dynamic simulation and multimodal data acquisition of the entire fruit harvesting process in unstructured scenarios. Furthermore, it can accurately characterize the damage mechanism and mechanical properties of the harvesting process based on the acquired data, effectively breaking through the bottleneck of traditional harvesting process data acquisition.

[0087] The detailed description of this invention is merely a specific description of feasible embodiments of this invention and is not intended to limit the scope of protection of this invention. All equivalent methods or modifications that do not depart from the technology of this invention should be included within the scope of protection of this invention.

Claims

1. A method for harvesting fruit in unstructured scenarios based on full-process biomimetic fruit data monitoring, characterized in that, include: Step 1) Construct a data monitoring bionic fruit with a multi-layer sensing structure. The outer layer of the data monitoring bionic fruit is used for force sensing and simulating the color of real fruit. Step 2) The end effector of the picking robot identifies, grasps and picks the data monitoring bionic fruit that presents the color of ripe fruit. At the same time, the data monitoring bionic fruit acquires grasping data in real time to characterize the physical response characteristics of the data monitoring bionic fruit in the whole picking process, thereby monitoring the whole picking process of the data monitoring bionic fruit in real time. In step 1), the multi-layer sensing structure of the data monitoring bionic fruit includes several epidermal sensing micro-units (1), an outer structural layer mesh unit (2), a force sensing frame unit (3), a cellular pulp unit (4), and a signal power supply structure unit (5) arranged sequentially from the outside to the inside, to simulate the upper epidermis, outer structural layer, lower epidermis, pulp layer, and inner structural layer of a real fruit in sequence. The outer structure of the data monitoring bionic fruit includes various epidermal sensing micro-units (1) used for contact sensing and simulating the color of a real fruit, and used to support the various epidermal sensing micro-units. The data monitoring bionic fruit also includes a shearing sensing unit to simulate the stem of a real fruit. The top of the signal power supply structure unit (5) is located at the top center of the shearing sensing unit installed on the top of the data monitoring bionic fruit, and the bottom of the shearing sensing unit is connected to the top of the signal power supply structure unit (5). Each epidermal sensing micro-element unit (1), the force sensing frame unit (3) and the shearing sensing unit are electrically connected to the signal power supply structure unit (5).

2. The unstructured fruit picking method based on whole-process biomimetic fruit data monitoring according to claim 1, characterized in that: The signal power supply structure unit (5) includes a communication module (502), a power management module (504), a microcontroller unit (MCU) module (505), an inertial measurement unit (IMU) module (506), an inner structural frame (507), and a wireless charging module. The inner structural frame (507) is integrally formed from a hollow spherical main frame and two central axis channels (501). The two central axis channels (501) are vertically arranged at the top center and bottom center of the spherical main frame, respectively. The top of the upper central axis channel (501) is connected to the bottom of the shear sensing unit. The outer periphery of the spherical main frame is covered with a second substrate layer. The communication module (502), power management module (504), microcontroller unit (MCU) module (505), inertial measurement unit (IMU) module (506), and wireless charging module are all integrated into the spherical main frame. The inertial measurement unit (IMU) module (506) and the wireless charging module are both installed inside the spherical main frame and isolated by several insulating partitions (508); the wireless charging module includes an inertial charging coil (509) and a coil base (510). The coil base (510) is installed in the lower part of the inner structural layer frame (507). The inertial charging coil (509) is wound on the coil base (510) and electrically connected to the power management module (504); the communication module (502), the power management module (504) and the inertial measurement unit (IMU) module (506) are all electrically connected to the microcontroller unit (MCU) module (505). The microcontroller unit (MCU) module (505) is electrically connected to each skin sensing micro-element unit (1), the force sensing frame unit (3) and the shear sensing unit.

3. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 2, characterized in that: The shear sensing unit includes two fruit stalk segments (6, 7). The first fruit stalk segment (6) is integrally formed from a ring hook (601) and a fruit stalk stem (602). The ring hook (601) is located at the top of the fruit stalk stem (602) to suspend the entire data monitoring bionic fruit in a fixed position. A first fruit stalk cavity (604) is opened in the bottom of the fruit stalk stem (602) and a permanent magnet (603) is installed inside it. A second fruit stalk cavity (704) is opened in the second fruit stalk segment (7). An excitation coil (701), an iron core, and a magnetic force meter (702) are installed in the second fruit stalk cavity (704). The excitation coil (701) is wound on the iron core, and the magnetic force meter... (702) Installed at one end of the iron core, the excitation coil (701) and the magnetic force meter (702) are electrically connected to the microcontroller MCU module (505); when the excitation coil (701) is energized, the end of the permanent magnet (603) is attracted to the other end of the iron core. At this time, the bottom end of the first fruit stalk segment (6) and the top end of the second fruit stalk segment (7) are connected and the two fruit stalk cavities (604, 704) are connected; when the bionic fruit is harvested, the separation of the permanent magnet (603) and the iron core is used to simulate the fruit stalk cutting process. The magnetic force feedback of the shearing force threshold at the moment of fruit stalk cutting is obtained by the magnetic force meter (702) when the permanent magnet (603) and the iron core are separated.

4. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 2, characterized in that: The force sensing frame unit (3) includes a flexible frame (301), several pressure sensor arrays (302) and several isolation layers (303). The flexible frame (301) is a hollow spherical structure. The signal power supply structure unit (5) is located in the center of the flexible frame (301). The top and bottom surfaces of the flexible frame (301) are provided with fruit stem interfaces (304) and are respectively fitted onto the upper and lower central axis channels (501) of the structural layer frame (507). Each isolation layer (303) is sealed and connected to the flexible frame (301) as the surface of the flexible frame (301) and does not cover the two fruit stem interfaces (304). Each pressure sensor array (302) is arranged on the outer surface of its own isolation layer (303). Each pressure sensor array (302) includes several pressure sensors arranged at uniform intervals and is electrically connected to the microcontroller unit (MCU) module (505). Each pressure sensor array (302) integrates the pressure data of its own pressure sensors and transmits them to the microcontroller unit (MCU) module (505).

5. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 4, characterized in that: The cellular pulp unit (4) is composed of a hydrogel array of several cellular structures. The cellular pulp unit (4) is filled between the second substrate layer of the inner structural frame (507) of the force perception frame unit (3) and the inner structural frame (507) of the signal power supply structure unit (5).

6. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 4, characterized in that: The outer structural layer mesh unit (2) is a spherical structure and a mesh envelope structure composed of several triangles. The flexible frame (301) of the force sensing frame unit (3) is connected to the interior of the outer structural layer mesh unit (2) but does not contact it. The overall frame of the outer structural layer mesh unit (2) is made of flexible tubes (201), and the connection position of each triangular frame composed of flexible tubes (201) serves as a self-positioning node (202). A microchip is installed on the inner side of each self-positioning node (202) through a plastic sheet. Each microchip has a different position. Each triangular frame is equipped with a skin sensing micro-element unit (1) and corresponds to a microchip and its position number. Each skin sensing micro-element unit (1) is electrically connected to the microcontroller MCU module (505). The top and bottom centers of the outer structural layer mesh unit (2) form a fruit stalk connection port (203) through the ends of several flexible tubes (201). The two fruit stalk connection ports (203) are respectively connected to the two central axis channels (501) of the structural layer frame (507).

7. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 6, characterized in that: Each of the aforementioned epidermal sensory micro-units (1) includes several tactile sensory micro-units (101), a first substrate layer (102), and an electrochromic layer (103). For each triangular frame composed of a flexible tube (201), the first substrate layer (102) is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit (2). The electrochromic layer (103) is attached to the outer surface of the first substrate layer (102), and each electrochromic layer (103) completely encloses the outer structural layer mesh unit (2). As an outer surface, the inner side of the first substrate layer (102) is uniformly spaced with tactile sensing micro-elements (101). Each tactile sensing micro-element (101) faces the pressure sensor array (302) but does not contact each other. Each tactile sensing micro-element (101) and the electrochromic layer (103) are electrically connected to the microcontroller unit (MCU) module (505). The microcontroller unit (MCU) module (505) applies voltage to the electrochromic layer (103) to change color according to the position number of different microchips.

8. The unstructured scene fruit picking method based on whole-process biomimetic fruit data monitoring according to claim 7, characterized in that: In step 2), the harvesting process of the data-monitored bionic fruit is divided into an identification stage, a harvesting stage, and a collection stage. In the identification stage, the microcontroller unit (MCU) module (505) applies voltage to each electrochromic layer (103) according to the position number of the microchip on the self-positioning node (202) to simulate the ripe color of real fruit. When the harvesting robot recognizes that the electrochromic layer (103) presents the ripe color of real fruit, it controls the end effector to grab the data-monitored bionic fruit and enter the harvesting stage. In the harvesting stage, when the end effector touches each tactile sensing micro-element (101), and when the tactile sensing micro-element (101) produces a piezoresistive change, it is determined that the end effector has been harvested. Contact is made to achieve tactile perception. The end effector continues to grasp until several pressure sensor arrays (302) generate pressure data to achieve pressure perception. Finally, the permanent magnet (603) and the iron core are separated. The shear force at the moment of separation is obtained by the magnetic force meter (702) to achieve shear perception. Then, the collection stage begins. In the collection stage, the end effector grasps the picked data monitoring bionic fruit and moves it to the placement position directly above the fruit basket. Finally, the end effector is released to put the data monitoring bionic fruit into the fruit basket. The inertial measurement unit (IMU) module (506) obtains the acceleration data of the data monitoring bionic fruit in real time during the movement and fall of the end effector to achieve motion perception.

9. The unstructured scene fruit picking method based on whole-process bionic fruit data monitoring according to claim 8, characterized in that: Before the identification stage, the data monitoring bionic fruit is first suspended by a ring hook (601), and the excitation coil (701) is energized by the microcontroller unit MCU module (505) to apply a preset adsorption force between the permanent magnet (603) and the iron core.

Images