Unstructured scene fruit picking method based on whole-process bionic fruit data monitoring

Abstract

The invention discloses a non-structural scene fruit picking method based on whole-process bionic fruit data monitoring. The method comprises the following steps: constructing a bionic fruit with a multi-layer sensing structure, wherein the outer layer structure of the bionic fruit is used for force sensing and simulating the color of a real fruit; then, the bionic fruit presenting the real fruit ripe color is recognized, grabbed and picked through an end effector of the picking robot, meanwhile, the bionic fruit obtains grabbing data in real time, and therefore the whole picking process of the bionic fruit is monitored in real time. According to the method, a robot-fruit-environment interaction application mode and a machine-fruit closed-loop interaction mode are provided, fruit picking full-process dynamic simulation and multi-modal data acquisition in a non-structural scene can be realized, a damage mechanism and mechanical properties in a picking process can be accurately represented, the data acquisition bottleneck in a traditional picking process is broken through, and the method is suitable for large-scale popularization and application. And a new path is provided for agricultural data acquisition.

Description

Technical Field

[0001] The invention relates to a fruit picking method, in particular to a non-structured scene fruit picking method based on full-process bionic fruit data monitoring. Background Art

[0002] As a high-value agricultural product, fruit occupies an important position in terms of market size, economic value and supply chain demands. Fruit picking is an important link in the fruit production process, and the cutting-edge exploration and industrial research and development of related technologies have always been a hot topic in the field of agricultural engineering.

[0003] In recent years, with the development and maturity of artificial intelligence (AI) technology, the application of intelligent robotics in agricultural scenarios has made significant progress. The resulting rise of embodied agricultural intelligence has also gradually entered the public eye. However, the lack of real-world data has been a significant constraint in the application and promotion of these technologies.

[0004] In the development of fruit harvesting technology, researchers are committed to finding automated fruit harvesting methods that are efficient, minimize damage, and have high success rates. However, these methods rely heavily on obtaining fruit information, such as its location, maturity, obstruction, size, shape, and orientation. This information collectively determines the accuracy, efficiency, and effectiveness of the harvesting operation.

[0005] There are two main data acquisition methods for existing fruit picking technology: the traditional picking method based on real fruit picking, and the picking method based on static simulation models. These two methods have the following problems: Traditional real fruit picking: data is difficult to obtain, information is difficult to quantify, and the process cannot be repeated. Static simulation model picking: simulation results are limited, the simulation process is single, and data cannot be collected. These issues directly or indirectly lead to high missed picking rates, low accuracy, and low effectiveness in the fruit picking process, making it difficult to simultaneously address the comprehensive needs of multiple dimensions such as fruit damage, picking efficiency, and complete picking when promoting the application of this technology.

[0006] In summary, in a complex, dynamic, and unstructured orchard environment, it is difficult to obtain the key multidimensional data and information required for the research, development, and application promotion of fruit picking technology in real time, accurately, and at low cost to solve the existing problems of picking efficiency, precision, and effectiveness. Summary of the Invention

[0007] To address the problems in the background art, the present invention provides a method for fruit picking in unstructured scenarios based on full-process bionic fruit data monitoring. This method can address the problems of low picking efficiency, insufficient precision, high damage rate, and poor effectiveness caused by the difficulty in acquiring key data during the research, development, and application promotion of artificial intelligence and robotics technologies. It can achieve unstructured fruit picking in unstructured scenarios driven by full-process, digital, and multifunctional data monitoring of bionic fruits. The method is driven by a set of bionic fruits that can achieve full-process, digital, and multifunctional data monitoring. The bionic fruits can collect, store, record, and wirelessly transmit multidimensional information data such as fruit color, position, movement, and damage required during the picking process in real time. The present invention can simulate and emulate the entire process of fruit picking, including identification, picking, and collection, effectively meeting the simulation and data collection needs throughout the entire cycle of scientific research, application development, and product promotion. It provides key technical support for resolving data acquisition bottlenecks in agricultural scenarios, particularly in the fruit picking process, and for improving the level of automation and intelligence in this field.

[0008] The technical solution adopted in the present invention is: The non-structured scene fruit picking method based on full-process bionic fruit data monitoring of the present invention comprises: Step 1) Construct a data monitoring bionic fruit with a multi-layer sensing structure. The outer structure of the data monitoring bionic fruit is used for force sensing and simulating the color of real fruit.

[0009] Step 2) The end effector of the picking robot identifies, grasps, and picks the data-monitoring bionic fruit that presents the ripe color of real fruit. At the same time, the data-monitoring bionic fruit obtains grasping data in real time to characterize the physical response characteristics of the data-monitoring bionic fruit throughout the picking process. The entire picking process of the data-monitoring bionic fruit is monitored in real time to obtain key multidimensional data and information during the fruit picking process, thereby improving picking efficiency and accuracy while achieving non-destructive picking.

[0010] In the step 1), the multi-layer perception structure of the data monitoring bionic fruit includes a number of epidermal perception micro-element units, an outer structure layer mesh unit, a force perception framework unit, a cellular structure flesh unit and a signal energy supply structure unit arranged in sequence from the outside to the inside, so as to simulate the upper epidermal layer, outer structure layer, lower epidermal layer, flesh layer and inner structure layer of a real fruit in sequence, namely, the outer peel, middle peel, inner peel and core, and balance the requirements of authenticity and functionality; the outer layer structure of the data monitoring bionic fruit includes various epidermal perception micro-element units for contact perception and simulating the color of the real fruit , used to support the outer structural layer mesh unit of each epidermal sensing micro-unit and the force sensing frame unit for sensing grasping pressure; the data monitoring bionic fruit also includes a shear sensing unit to simulate the fruit stem of a real fruit, and the top of the signal power supply structure unit is located at the top center of the data monitoring bionic fruit and is located outside each epidermal sensing micro-unit. The bottom of the shear sensing unit is connected to the top of the signal power supply structure unit; each epidermal sensing micro-unit, force sensing frame unit, and shear sensing unit are all electrically connected to the signal power supply structure unit. The data monitoring bionic fruit is used to simulate the color changes, touch, pressure, position, and motion path of the fruit involved in the identification, grasping, and collection stages during fruit picking, and obtain the corresponding data in real time.

[0011] The signal energy supply structure unit includes a communication module, an energy supply management module, a microcontroller unit MCU module, an inertial measurement unit IMU module, an inner structure layer frame and a wireless charging module. The inner structure layer frame is formed by a hollow spherical main frame and two central axis channels to simulate the core of a fruit. The two central axis channels are vertically arranged at the top center and the bottom center of the spherical main frame respectively. The top of the upper central axis channel is connected to the bottom end of the shear sensing unit. The outer peripheral surface of the spherical main frame is covered with a second substrate layer; the communication module, the energy supply management module, the microcontroller unit MCU module, the inertial measurement unit IMU module and the wireless charging module are all installed inside the spherical main frame and isolated by a number of insulating partitions; the wireless charging module includes an infinite charging coil and a coil base. The coil base is installed in the lower part of the inner structure layer frame, the infinite charging coil is wound on the coil base and electrically connected to the energy supply management module to supply energy to the internal module through wireless charging; the communication module, energy supply management module and inertial measurement unit IMU module are all electrically connected to the micro control unit MCU module through electrical connecting lines, and the micro control unit MCU module is electrically connected to each epidermal sensing micro-element unit, force sensing frame unit and shear sensing unit through the interlayer electrical connecting lines located in the two central axis channels. The interlayer electrical connecting lines are mainly used for data transmission, signal control and power supply between the various layers of the simulated fruit and the signal energy supply structure units, as well as mechanical and electrical connections with the shear sensing unit.

[0012] The shear sensing unit includes two fruit stalk segments, the first fruit stalk segment is formed as a whole by a ring hook and a fruit stalk stem, the ring hook is located at the top of the fruit stalk stem to hang the data monitoring bionic fruit as a whole in a fixed position, a first fruit stalk cavity is opened in the bottom of the fruit stalk stem and a permanent magnet is installed inside, a second fruit stalk cavity is opened inside the second fruit stalk segment, an excitation coil, an iron core and a magnetic force meter are installed in the second fruit stalk cavity, the excitation coil is wound on the iron core, the magnetic force meter is installed at one end of the iron core, the excitation coil and the magnetic force meter are electrically connected to the microcontroller through signal connection lines and interlayer electrical connection lines Unit MCU module; the excitation coil can be energized to apply different adsorption forces according to the relationship between different fruit types and maturity and magnetic attraction. When the excitation coil is energized, the end of the permanent magnet is adsorbed to the other end of the iron core. At this time, the bottom end of the first fruit stalk segment and the top end of the second fruit stalk segment are connected and the two fruit stalk cavities are connected; when the bionic fruit is picked, the data monitoring is carried out to simulate the fruit stalk cutting process by separating the permanent magnet and the iron core. The magnetic attraction force when the permanent magnet and the iron core are separated is obtained through the magnetic attraction meter to feedback the shear force threshold at the moment of fruit stalk cutting to provide a basis for the actual picking process.

[0013] The force perception frame unit includes a flexible frame, several pressure sensor arrays and several isolation layers. The flexible frame is a hollow spherical structure. The signal power supply structure unit is located in the inner center of the flexible frame. The top and bottom centers of the flexible frame are both provided with fruit stalk interfaces and are respectively rotated on the upper and lower center axis channels of the structural layer frame. Several electrical interfaces are provided in the circumference of the fruit stalk interface and the center axis channel for passing electrical connecting lines; each isolation layer is sealed and connected to the flexible frame as the surface of the flexible frame and does not cover the two fruit stalk interfaces. Each pressure sensor array is arranged on the outer surface of each isolation layer. No pressure sensor array is arranged on the isolation layer near the two fruit stalk interfaces. Each pressure sensor array includes several pressure sensors arranged evenly spaced and is electrically connected to the microcontroller unit MCU module through the electrical interface and electrical connecting lines set in the isolation layer. Each pressure sensor array integrates the pressure data of its own pressure sensors and transmits them to the microcontroller unit MCU module.

[0014] The cellular structure pulp unit is composed of several layers of cellular structure hydrogel arrays, and the cellular structure pulp unit is filled between each isolation layer of the force sensing frame unit and the second substrate layer of the inner structure layer frame of the signal energy supply structure unit.

[0015] The outer structural layer mesh unit is a spherical structure and a mesh envelope structure composed of several triangles. The flexible frame of the force perception frame unit is in contact with the interior of the outer structural layer mesh unit without contact. The entire frame of the outer structural layer mesh unit is made of soft tubes, and the connection positions of each triangular frame composed of soft tubes serve as self-positioning nodes. The soft tubes are mainly used to control the arrangement of signal lines, data signal lines, and power supply lines. A microchip is installed on the inner side of each self-positioning node through a thin plastic sheet. Each microchip has a different position number and is electrically connected to the microcontroller unit (MCU module). Each triangular frame is installed with an epidermal perception microelement unit and corresponds to a respective microchip and its position number. Each epidermal perception microelement unit is electrically connected to the microcontroller unit (MCU module). The top and bottom centers of the outer structural layer mesh unit are formed by the ends of several soft tubes to form a fruit handle connection port. The two fruit handle connection ports are respectively connected to the two central axis channels of the structural layer frame. The fruit handle connection port is also used to converge the control signal line, data signal line, and power supply line into the central axis channel to achieve electrical connection with the microcontroller unit (MCU module). The main function of the mesh unit in the outer structural layer is to provide mechanical support and electrical connection for the surface of the simulated fruit, so that the epidermal sensing micro-units can fit effectively.

[0016] Each of the epidermal sensing micro-element units includes a number of tactile sensing micro-elements, a first substrate layer and an electrochromic layer. For each triangular frame composed of soft tubes, the first substrate layer is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit. The electrochromic layer is attached to the outer side of the first substrate layer. Each electrochromic layer completely wraps the outer structural layer mesh unit and serves as the outer surface. The inner side of the first substrate layer is evenly spaced with a number of tactile sensing micro-elements. Each tactile sensing micro-element is facing the pressure sensor array and does not contact each other. Each tactile sensing micro-element and the electrochromic layer are electrically connected to the microcontroller unit MCU module. The microcontroller unit MCU module applies voltage to the electrochromic layer to change color according to the position number of different microchips.

[0017] In the step 2), the data monitoring bionic fruit picking process is divided into an identification stage, a picking stage and a collection stage. In the identification stage, the microcontroller unit MCU module applies a voltage to each electrochromic layer according to the position number of the microchip on the self-positioning node to simulate the ripe color of the real fruit. When the picking robot recognizes that the electrochromic layer presents the ripe color of the real fruit, it controls the end effector to grab the data monitoring bionic fruit and enter the picking stage; in the picking stage, when the end effector touches each tactile sensing micro-element and the tactile sensing micro-element produces a piezoresistive change, it is determined that the end effector has been in contact to achieve Tactile perception: The end effector continues to grasp until several pressure sensor arrays generate pressure data to achieve pressure perception, and finally separates the permanent magnet and the iron core. The shear force at the moment of separation is obtained through the magnetic force meter to achieve shear perception, and then enters the collection phase. During the collection phase, the end effector grasps the picked data-monitoring bionic fruit and moves it to the placement position directly above the fruit basket. Finally, the end effector is released to place the data-monitoring bionic fruit into the fruit basket. The inertial measurement unit (IMU) module obtains real-time acceleration data of the data-monitoring bionic fruit as it moves and falls with the end effector to achieve motion perception.

[0018] Before the identification stage, the data monitoring bionic fruit is first hung through a ring hook, and the excitation coil is energized through the microcontroller unit MCU module to apply a preset adsorption force between the permanent magnet and the iron core.

[0019] The beneficial effects of the present invention are: 1) Dynamic simulation and multimodal data collection of the entire fruit picking process in unstructured scenarios: This invention covers three key stages: identification, picking, and collection. During the identification stage, electrochromic technology can be used to dynamically simulate fruit color changes. During the picking stage, tactile and pressure sensors can be integrated to simulate the clamping force state. Magnetic force calculations can be used to accurately simulate shear forces, enabling real-time capture of picking movements and position changes. During the collection stage, changes in fruit placement can be continuously tracked.

[0020] 2) Accurately characterize damage mechanisms and mechanical properties during the harvesting process: This invention can effectively quantify and record key parameters such as touch, force, and pressure. By summarizing and analyzing multi-source data, and using the mapping relationship between fruit damage mechanisms and mechanical properties obtained from external reference instruments and the characteristics of sensor simulation signals as reference, it generates a visual damage and force characterization map, filling the gap in dynamic, in-situ data acquisition and characterization in this field.

[0021] 3) Breaking through the bottleneck of traditional data acquisition during the harvesting process: This invention overcomes the difficulties of data collection in real unstructured environments or static models, providing a new path for agricultural data acquisition. It also pioneers a robot-fruit-environment interaction model in which sensors are external to the collection system, as well as a new closed-loop method for machine-fruit interaction.

[0022] 4) Provide highly customizable fruit simulation: The present invention can flexibly adjust fruit growth parameters such as color, firmness, and stalk characteristics according to research needs, and realize customized design in terms of weight, size, shape, etc., to provide full-process data for the real picking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the overall structure of the data monitoring bionic fruit provided by an embodiment of the present invention; Figure 2 A schematic diagram of the physical layers of the data monitoring bionic fruit provided by an embodiment of the present invention; Figure 3 Schematic diagram of the epidermal sensing micro-element unit for data monitoring bionic fruit provided by an embodiment of the present invention; Figure 4 A schematic diagram of the mesh unit of the outer structure layer of the data monitoring bionic fruit provided by an embodiment of the present invention; Figure 5 A schematic diagram of a force perception framework unit for a data monitoring bionic fruit provided by an embodiment of the present invention; Figure 6 A schematic diagram of the cellular structure and pulp layer of a data-monitoring bionic fruit provided in an embodiment of the present invention; Figure 7 A structural diagram of the signal energy supply structure unit of the data monitoring bionic fruit provided by an embodiment of the present invention; Figure 8 A three-dimensional diagram of the signal energy supply structure unit of the data monitoring bionic fruit provided by an embodiment of the present invention; Figure 9 A cross-sectional view of a shear sensing unit of a data monitoring bionic fruit provided in an embodiment of the present invention; Figure 10 A diagram of the system architecture for data monitoring bionic fruit provided by an embodiment of the present invention; Figure 11 This is a diagram of the harvesting process simulation process provided by an embodiment of the present invention; Figure 12 A flowchart of a method for picking fruit in an unstructured scene based on bionic fruit data monitoring and driving provided by an embodiment of the present invention; Figure 13 A texture analyzer reference data diagram of the picking process provided by an embodiment of the present invention; Figure 14 A graph showing pressure data of a capacitive flexible sensor during the picking process provided by an embodiment of the present invention; Figure 15 This is an acceleration simulation curve diagram of the picking process provided by an embodiment of the present invention, wherein: Figure 15 (a) is the acceleration simulation curve when the robot arm approaches the fruit. Figure 15 (b) is the acceleration simulation curve of the robot arm placement motion link. Figure 15 (c) is a curve diagram of the acceleration simulation of the robot arm collecting and putting into the basket; In the figure: 1. Epidermal sensing microelement unit, 101. Tactile sensing microelement, 102. First substrate layer, 103. Electrochromic layer, 2. Outer structure layer mesh unit, 201. Soft tube, 202. Self-positioning node, 203. Fruit stalk connection port, 3. Force sensing frame unit, 301. Flexible frame, 302. Pressure sensor array, 303. Isolation layer, 304. Fruit stalk interface, 4. Cellular structure pulp unit, 5. Signal energy supply structure unit, 501. Central axis channel, 502. Communication module, 503. Electrical connection line, 504. 04. Energy supply management module, 505. Microcontroller unit MCU module, 506. Inertial measurement unit IMU module, 507. Inner structure layer frame, 508. Insulating partition plate, 509. Infinite charging coil, 510. Coil base, 511. Interlayer electrical connection line, 6. First fruit stalk segment, 601. Ring hook, 602. Fruit stalk stem, 603. Permanent magnet, 604. First fruit stalk cavity, 7. Second fruit stalk segment, 701. Excitation coil, 702. Magnetic force meter, 703. Signal connection line, 704. Second fruit stalk cavity. DETAILED DESCRIPTION

[0024] To facilitate understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Specific embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive disclosure of the present invention.

[0025] The non-structured scene fruit picking method based on full-process bionic fruit data monitoring of the present invention is as follows: Step 1) Construct a data monitoring bionic fruit with a multi-layer sensing structure. The outer structure of the data monitoring bionic fruit is used for force sensing and simulating the color of real fruit. The data monitoring bionic fruit used in the embodiment of the present invention to drive the unstructured scene fruit picking method mainly includes two parts: a simulated fruit stalk and a simulated fruit. The simulated fruit stalk is a curved hollow cylindrical structure made of polylactic acid (PLA) hard plastic. The simulated fruit is a multi-layer structure with an outer layer of flexible material and an inner layer of hard material. Figure 1 and Figure 2 As shown, the multi-layer perception structure of the data monitoring bionic fruit includes a number of epidermal perception micro-element units 1, outer structure layer mesh units 2, force perception framework units 3, cellular structure flesh units 4 and signal energy supply structure units 5 arranged in sequence from the outside to the inside, so as to simulate the upper epidermal layer, outer structure layer, lower epidermal layer, flesh layer and inner structure layer of real fruit in sequence, namely the outer peel, middle peel, inner peel and core, and balance the requirements of authenticity and functionality. In this embodiment, the main layers of the simulated fruit multi-layer structure are: the upper epidermal layer with the epidermal perception micro-element unit 1 as the main functional structure, the outer structure layer with the outer structure layer mesh unit 2 as the main functional structure, the lower epidermal layer with the force perception framework unit 3 as the main functional structure, the flesh layer with the cellular structure flesh unit 4 as the main functional structure, and the signal and energy supply layer with the signal energy supply structure unit 5 as the main functional structure. The upper epidermis layer is attached to the outer structure layer, and there is a gap of about 1-2mm between the upper epidermis layer and the lower epidermis layer to avoid triggering the pressure sensor of the lower epidermis layer. The pulp layer is attached to the inner surface of the lower epidermis layer and extends to the outside of the inner structure layer. The inner structure layer and the lower epidermis layer jointly provide support for the pulp layer. The signal and energy supply layer are distributed in the inner layer of the inner structure layer and are located in the core of the simulated fruit. Specifically, in this embodiment, there is a gap of about 1mm between the force perception frame unit 3 and the outer structure layer mesh unit 2 to avoid false triggering due to self-extrusion between the tactile perception micro-element 101 and the pressure sensor array 302. The outer structure layer mesh unit 2 is enclosed on the outside of the force perception frame unit 3, and the epidermis perception micro-element unit 1 is attached to its surface. The cellular structure pulp unit 4 is attached to the inner side of the isolation layer 303 of the force perception frame unit 3 and extends to the outside of the inner structure layer frame 507 of the signal energy supply structure unit 5. The figure is only a schematic representation.

[0026] The outer structure of the data-monitoring bionic fruit includes individual epidermal sensing micro-units 1 for contact sensing and simulating the color of real fruit, an outer structural layer mesh unit 2 for supporting each epidermal sensing micro-unit 1, and a force sensing framework unit 3 for sensing grasping pressure. The data-monitoring bionic fruit also includes a shear sensing unit to simulate the stem of a real fruit. The top of the signal power supply structure unit 5 is located at the top center of the data-monitoring bionic fruit and is located outside each epidermal sensing micro-unit 1. The bottom of the shear sensing unit is connected to the top of the signal power supply structure unit 5. Each epidermal sensing micro-unit 1, force sensing framework unit 3, and shear sensing unit are all electrically connected to the signal power supply structure unit 5. The data-monitoring bionic fruit is used to simulate the color changes, touch, pressure, position, and motion path of the fruit involved in the identification, grasping, and collection stages of the fruit picking process, and to acquire the corresponding data in real time.

[0027] like Figure 7 and Figure 8 As shown, the signal energy supply structure unit 5 includes a communication module 502, an energy supply management module 504, a micro control unit MCU module 505, an inertial measurement unit IMU module 506, an inner structure layer frame 507 and a wireless charging module. The inner structure layer frame 507 is formed as a whole by a hollow spherical main frame and two central axis channels 501 to simulate the core of a fruit. The two central axis channels 501 are vertically arranged at the top center and the bottom center of the spherical main frame respectively. The two central axis channels 501 are hollow tubular structures made of hard plastic or hard rubber. The top of the upper central axis channel 501 is connected to the shear sensing unit. At the bottom, the outer peripheral surface of the spherical main frame is covered with a second substrate layer, which is specifically made of transparent latex material; the communication module 502, the energy supply management module 504, the microcontroller unit MCU module 505, the inertial measurement unit IMU module 506 and the wireless charging module are all installed inside the spherical main frame and isolated by a number of insulating partitions 508. Constrained by the weight of the entire fruit, the inner structure layer frame 507 and the insulating partition 508 need to have a certain hardness and not be too heavy. Therefore, the inner structure layer frame 507 is specifically made of a hard low-density alloy with a thickness of 5-10 mm, such as a magnesium-lithium alloy or a carbon fiber material, and the insulating partitions 508 are made of a hard low-density alloy with a thickness of 5-10 mm. The partition 508 is specifically made of a hard low-density alloy material such as magnesium alloy and an insulating film. The center of the insulating partition 508 is hollowed out to pass the electrical connection line 503. The wireless charging module includes a wireless charging coil 509 and a coil base 510. The coil base 510 is installed in the lower part of the inner structural layer frame 507. The wireless charging coil 509 is wound on the coil base 510 and electrically connected to the energy supply management module 504 to supply energy to the internal module through wireless charging. The communication module 502, the energy supply management module 504 and the inertial measurement unit IMU module 506 are all electrically connected to the micro control unit MCU module 505 via the electrical connection line 503. The communication module 502 can use a WIFI module and a Bluetooth module to wirelessly transmit the data of the microcontroller unit MCU module 505 to an external computer device, and the energy supply management module 504 can use a battery protection board for power supply; the microcontroller unit MCU module 505 is electrically connected to each epidermal sensing micro-element unit 1, force sensing frame unit 3 and shear sensing unit through the interlayer electrical connection line 511 located in the two central axis channels 501. The interlayer electrical connection line 511 is mainly used for data transmission, signal control and power supply between each layer of the simulated fruit and the signal energy supply structure unit 5, as well as mechanical and electrical connection with the shear sensing unit.

[0028] In this embodiment, the materials selected for the inner structure layer frame 507 and the insulating partition plate 508 are hard low-density alloy materials, magnesium-lithium alloys. In addition, carbon fiber materials can also be selected as component materials. The surface of the insulating partition plate 508 is wrapped with an insulating film to prevent the alloy or carbon fiber material from directly contacting the components and thus conducting electricity. The entire inner structure layer is separated by a number of insulating partition plates 508. In this embodiment, the insulating partition plates 508 divide the inner structure layer frame 507 into 4 layers, which are used to install different circuit modules respectively. The center is hollowed out for passing electrical connection lines. The insulating partition plate 508 is circular in shape, and the arc ends of the inner structure layer frame 507 are respectively connected to the central axis channel 501. The coil base 510 is made of polypropylene plastic material, which is mechanically connected to the bottom of the lowest insulating partition plate 508, and the middle is also a hollow structure. The wireless charging coil 509 spirals downward from the junction of the coil base 510 and the separator 508, tightly wrapping around the outer surface of the coil base 510. It is directly connected to the energy supply management module 504 via an electrical connection 503, located in the hollow structure in the center of the coil base 510. The coil diameter of the wireless charging coil 509 is approximately 44 mm. The energy supply management module 504 is powered by the electrical connection 503 and is mounted on the upper surface of the bottommost insulating separator 508. Specifically, the electrical connection 503 connects the upper and lower adjacent layers through the hollow center of the insulating separator 608, facilitating signal communication between the modules in the energy supply layer. In this embodiment, the charge and discharge management module utilizes an 18650 lithium battery integrated charge and discharge module. The microcontroller unit 505, comprising a controller circuit and STM32F407 chip, is mounted on the upper surface of the middle insulating separator 508. It offers high performance and low power consumption, enabling it to effectively perform tasks such as signal control and data acquisition. Communication module 502 utilizes Wi-Fi and Bluetooth modules, also mounted on the top surface of the middle insulating partition 508, but symmetrically positioned with microcontroller unit 505. The Wi-Fi module primarily functions for data transmission, while the Bluetooth module primarily facilitates communication between the fruit and other fruits. Inertial measurement unit (IMU) module 506 utilizes the MMA8452Q and is mounted on the top surface of the top insulating partition 508. Its compact size and low power consumption make it well-suited for miniaturized designs.

[0029] like Figure 9As shown, the shear sensing unit includes two curved hollow cylindrical fruit stem segments 6 and 7, which are used to simulate the fruit stem of a real fruit and simulate the picking force when picking the fruit and collect shear force data. Acid) or ternary copolymer ABS (Acrylonitrile-Butadiene-Styrene) plastic, the first fruit stalk segment 6 is formed as a whole by a ring hook 601 and a fruit stalk stem 602, the ring hook 601 is located at the top of the fruit stalk stem 602 to hang the data monitoring bionic fruit as a whole in a fixed position, a first fruit stalk cavity 604 is opened in the bottom of the fruit stalk stem 602 and a permanent magnet 603 is installed inside, the permanent magnet 603 can be made of aluminum nickel cobalt alloy, a second fruit stalk cavity 704 is opened in the second fruit stalk segment 7, an excitation coil 701, a cylindrical iron core and a magnetic dynamometer 702 are installed in the second fruit stalk cavity 704, the excitation coil 701 is wound on the iron core, and the magnetic dynamometer 702 is installed at one end of the iron core The excitation coil 701 and the magnetic attraction meter 702 are electrically connected to the microcontroller unit MCU module 505 via the signal connection line 703 and the interlayer electrical connection line 511. The excitation coil 701 can be energized to apply different adsorption forces based on the relationship between different fruit types and maturity and the magnetic attraction force. When the excitation coil 701 is energized, the end of the permanent magnet 603 is attracted to the other end of the iron core. At this time, the bottom end of the first fruit stalk segment 6 and the top end of the second fruit stalk segment 7 are connected, and the two fruit stalk cavities 604 and 704 are connected. When the bionic fruit is picked, the data monitoring simulates the fruit stalk cutting process by separating the permanent magnet 603 and the iron core. The magnetic attraction force when the permanent magnet 603 and the iron core are separated is obtained by the magnetic attraction meter 702, and the shear force threshold at the moment of fruit stalk cutting is fed back to provide a basis for the actual picking process. In this embodiment, the curved hollow cylindrical structure simulated fruit stalk is mainly divided into two parts, namely the first fruit stalk segment 6 and the second fruit stalk segment 7. The annular hook 601, the stalk stem 602, the first stalk cavity 604, and the second stalk segment 7 of the first stalk segment 6 are all hollow structures. The second stalk cavity 704 of the second stalk segment 7 is made of PLA plastic by 3D printing. The permanent magnet 603 of the first stalk segment 6 can be made of aluminum-nickel-cobalt alloy. The excitation coil 701 of the second stalk segment 7 is mainly used to generate magnetic forces of different sizes. The magnetic force meter 702 is mainly used to detect changes in shear force during shearing or grasping and collect data. The signal connection line 703 is used to transmit control signals, data signals and power supply. The permanent magnet 603 of the first stalk segment 6 and the excitation coil 701, magnetic force meter 702, and signal connection line 703 of the lower section of the stalk constitute a stalk cutting sensor for feedback of shear force.

[0030] like Figure 5As shown, the force perception frame unit 3 includes a flexible frame 301, a plurality of pressure sensor arrays 302 and a plurality of isolation layers 303. The flexible frame 301 is a hollow spherical structure and adopts a solid rubber column or a solid flexible columnar material. The hollow structure is similar to the longitude and latitude distribution. The signal energy supply structure unit 5 is located in the inner center of the flexible frame 301. The top and bottom centers of the flexible frame 301 are provided with a fruit handle interface 304 and are respectively rotated on the upper and lower center axis channels 501 of the structural layer frame 507. The fruit handle interface 304 and the center axis channel 501 are provided with a plurality of circumferential directions. The electrical interface is connected through the electrical connection line; each isolation layer 303 is sealed and connected to the flexible frame 301 as the surface of the flexible frame 301 and does not cover the two fruit handle interfaces 304. The isolation layer 303 uses a flexible material with a certain thickness, such as a rubber material. Each pressure sensor array 302 is arranged on the outer surface of each isolation layer 303. No pressure sensor array is arranged on the isolation layer 303 near the two fruit handle interfaces 304. Each pressure sensor array 302 includes a number of evenly spaced pressure sensors and is electrically connected to the microcontroller unit MCU module 505 through the electrical interface and electrical connection line provided in the isolation layer 303. Each pressure sensor array 302 integrates the pressure data of each pressure sensor of itself and transmits it to the microcontroller unit MCU module 505. The pressure sensor can be a strain gauge pressure sensor or a piezoresistive pressure sensor; the flexible frame 301 mainly provides mechanical support for the entire lower epidermis, and can simulate the pressure feedback of fruits with a certain hardness; the pressure sensor array 302 is used to collect relevant data on the pressure position and pressure intensity, which is mainly tightly attached to the isolation layer through a certain physical and chemical process. The isolation layer 303 is arranged in an array shape, except for the part close to the fruit stem interface 304, which is not distributed, and the other areas are evenly distributed; in addition to being used to attach the pressure sensor array 302, the isolation layer 303 also bears the role of burying the control signal line, data signal line and power supply line, and converges the control signal line, data signal line and power supply line to the fruit stem interface 304 on the upper and lower sides to achieve electrical connection with the micro control unit MCU module 505, completely wrapping the entire lower epidermis of the fruit; the fruit stem interface 304 is also used to install and fix the central axis channel 501 to fix the entire simulated fruit. In this embodiment, the force perception frame unit 3 is a spherical frame with a diameter of 95mm. The main supporting structure of the frame is a flexible frame 301, the main component of which is a solid rubber column, which can provide mechanical support for the overall structure. With the frame structure as the skeleton, an isolation layer 303 is also distributed on the force perception frame unit 3. The isolation layer 303 is mainly composed of rubber material. The part of its upper surface that contacts the pressure sensor array 302 is distributed with electrical interfaces that can be connected to the pressure sensor. Control signal lines, data signal lines and power supply lines are distributed inside and arranged according to a certain pattern.With the horizontal central axis as the dividing line, electrical connections above the axis converge to the upper stem interface 304, while those below the axis converge to the lower stem interface 304. The stem interface 304 is a hollow, annular structure primarily composed of hard plastic or carbon fiber. Electrical connections are distributed around its perimeter, connecting all electrical connections of the force sensing frame unit 3 to the central axis channel 501, enabling data exchange and energy supply between the lower epidermis and the signal and energy supply layers. Furthermore, the stem interface 304 is used to mount and secure the simulated stem segments 6 and 7 and the central axis channel 501.

[0031] like Figure 6 As shown, the cellular structured fruit pulp units 4 are constructed using several layers of cellular hydrogel arrays, each layer comprising several rows and columns of cellular hydrogel. The cellular structured fruit pulp units 4 are interposed between the isolation layers 303 of the force sensing framework unit 3 and the second substrate layer of the inner structural layer frame 507 of the signal energy supply structure unit 5. Through specific physical and chemical processes, the cellular structured fruit pulp units 4 are tightly attached to the inside of the isolation layer 303 and horizontally distributed, extending all the way to the second substrate layer to simulate the texture of fruit pulp. The hydrogel can be a highly water-retaining natural polymer hydrogel or a synthetic gel, and the resulting array can be encapsulated using polydimethylsiloxane (PDMS) or thermoplastic polyurethane (TPU). In this embodiment, the cellular structure pulp unit 4 extends horizontally from the inner surface of the isolation layer 303 to the outside of the inner structure layer frame 507. The two structures provide mechanical support for it. At the same time, the outermost layer and the innermost layer of the cellular structure pulp unit 4 are adhered to the inner side of the isolation layer 303 and the outer side of the inner structure layer frame 507 respectively.

[0032] like Figure 4As shown, the outer structure layer mesh unit 2 is a spherical structure and a mesh envelope structure composed of several triangles. The flexible frame 301 of the force perception frame unit 3 is inscribed in the inner part of the outer structure layer mesh unit 2 and does not touch each other; the overall frame of the outer structure layer mesh unit 2 adopts a soft tube 201, and the soft tube 201 adopts a colloid material, and the connection position of each triangular frame composed of the soft tube 201 serves as a self-positioning node 202. The soft tube 201 is mainly used to control the arrangement of signal lines, data signal lines and power supply lines. A microchip is installed on the inner side of each self-positioning node 202 through a thin plastic sheet. The microchip adopts a decoder chip and a circuit. The circuit can select each epidermal perception micro-element unit 1 by address selection. Each microchip has a different position number and is electrically connected The microcontroller unit MCU module 505 uses these address codes to accurately read and control the target epidermal sensing microunit 1. Each triangular frame is equipped with an epidermal sensing microunit 1 and corresponds to a corresponding microchip and its position number. Each epidermal sensing microunit 1 is electrically connected to the microcontroller unit MCU module 505. The top and bottom centers of the outer structural layer mesh unit 2 form a fruit stalk connection port 203 through the ends of several soft tubes 201. The two fruit stalk connection ports 203 are respectively connected to the two central axis channels 501 of the structural layer frame 507. The fruit stalk connection ports 203 are also used to converge the control signal line, data signal line, and power supply line into the central axis channel 501 to achieve electrical connection with the microcontroller unit MCU module 505. The main function of the outer structural layer mesh unit 2 is to provide mechanical support and electrical connection for the simulated fruit surface, so that the epidermal sensing microunit 1 can be effectively attached. The outer structural layer mesh units 2 are distributed in an envelope shape around the outside of the simulated fruit. In this embodiment, the outer structural layer mesh units 2 are formed by connecting five vertically parallel 12-sided polygons. All external connection points are distributed on a circle with a diameter of approximately 100 mm. Self-positioning nodes 202 are distributed at all external connection points. Self-positioning nodes 202 are equipped with a micro-decoder chip, its supporting circuitry, and a thin plastic sheet. Micro-interface circuits are also distributed at the connection between self-positioning nodes 202 and epidermal sensing micro-units 1, which are used to connect the electrical circuits of epidermal sensing micro-units 1. Each epidermal sensing micro-unit 1 corresponds to a self-positioning node 202.

[0033] like Figure 3As shown, each epidermal sensing micro-element unit 1 includes a number of tactile sensing micro-elements 101, a first substrate layer 102 and an electrochromic layer 103. For each triangular frame formed by the soft tube 201, the first substrate layer 102 is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit 2. The electrochromic layer 103 is attached to the outer side surface of the first substrate layer 102. Each electrochromic layer 103 completely wraps the outer structural layer mesh unit 2 and serves as the outer surface. The inner side surface of the first substrate layer 102 is evenly spaced with a number of tactile sensing micro-elements 101. Each tactile sensing micro-element 101 is facing the pressure sensor array 302 and does not contact each other. Each tactile sensing micro-element 101 and the electrochromic layer 103 are electrically connected to the microcontroller unit MCU module 505. The microcontroller unit MCU module 505 applies voltage to the electrochromic layer 103 to change color according to the position number of different microchips. The first substrate layer 102 has the same size as the triangular frame in which it is located, and the outer side surface is a plane, a convex curved surface or a triangular prism structure. The electrochromic layer 103 is completely attached to the outer side surface of the first substrate layer 102. When it is a triangular prism structure, the electrochromic layer 103 is divided into three structures and is completely attached to the three surfaces of the triangular prism respectively; the first substrate layer 102 can be made of a rubber material or a flexible film material of a certain thickness, such as polyethylene terephthalate PET and polydimethylsiloxane PDMS; the tactile sensing micro-element 101 can be made of a capacitive or resistive piezoelectric material, the characteristic of this type of material is that when it is under pressure, the electrical values ​​such as capacitance and resistance will change with the pressure; the electrochromic layer 1 03 uses inorganic electrochromic materials or organic electrochromic materials, such as tungsten trioxide WO3. Such materials can reversibly change optical properties under the condition of applied voltage; the tactile sensing micro-element 101 and the electrochromic layer 103 are integrated on the first substrate layer 102 through certain physical and chemical processes and reserve control signals, data signals and power supply interfaces. It can complete the simulation of the colors of fruits with different degrees of maturity in the fruit picking and identification link, and help verify the actual functions and recognition effects of external recognition equipment and recognition algorithms; and it can sense and collect data on the accidental touch of the fruit by the end effector of the picking mechanism in the fruit picking and grasping approach link, helping to improve the accuracy of the fruit picking and grasping approach link.During specific implementation, the epidermal sensing micro-element units 1 are densely distributed on the surface of the simulated fruit at an angle of 30°-60° to the tangent of the vertical surface; the tactile sensing micro-element 101 has a radius of about 1.2mm, a height of about 0.3mm, and an interval of about 0.3mm between adjacent micro-elements. It is internally integrated with capacitive piezoelectric material and is integrated on the smooth lower surface of the first substrate layer 102 through physical and chemical processes. The upper surface of the first substrate layer 102 is a triangular pyramid structure with a certain curvature made of rubber material, and its lower surface is a curved surface about 3-5mm thick, on which electrochromic material 103 is integrated, which is mainly 1-3mm thick micro-elements made of tungsten trioxide WO3, distributed on the surface of the triangular pyramid structure; electrical connection lines are also integrated inside the first substrate layer 102, and interfaces are reserved.

[0034] like Figure 10 As shown, the system architecture of the bionic fruit of the present invention mainly includes seven parts: the fruit stem, the upper epidermis, the outer structural layer, the lower epidermis, the flesh layer, the inner structural layer, and the signal and energy supply layer. The key components include the shear force receptor in the fruit stem, the tactile sensor microelement 101 and electrochromic material 103 in the upper epidermis, the frame composed of the soft tube 201 in the outer structural layer, the flexible frame 301, pressure sensor array 302, and insulating isolation layer 303 in the lower epidermis, the cellular structure flesh unit 4 in the flesh layer, the insulating partition plate 508 and inner structural layer frame 507 in the inner structural layer, and the communication module 502, microcontroller unit MCU module 505, inertial measurement unit IMU module 506, and wireless charging coil 509 in the signal and energy supply layer.

[0035] Step 2) The harvesting robot's end effector identifies, grasps, and picks a data-monitoring bionic fruit that displays the ripe color of a real fruit. Simultaneously, the data-monitoring bionic fruit acquires real-time grasping data to characterize the physical response characteristics of the data-monitoring bionic fruit throughout the harvesting process. This allows real-time monitoring of the entire harvesting process, capturing key multidimensional data and information about the fruit, thereby improving harvesting efficiency and accuracy while achieving non-destructive harvesting. The harvesting process of the data-monitoring bionic fruit is divided into the recognition phase, the harvesting phase, and the collection phase. Prior to the recognition phase, the data-monitoring bionic fruit is first suspended by a ring hook 601, and the microcontroller unit MCU module 505 energizes the excitation coil 701 to apply a predetermined attraction force between the permanent magnet 603 and the iron core.During the recognition phase, the microcontroller unit MCU module 505 applies voltage to each electrochromic layer 103 according to the position number of the microchip on the self-positioning node 202 to simulate the ripe color of the real fruit. When the picking robot recognizes that the electrochromic layer 103 presents the ripe color of the real fruit, it controls the end effector to grab the data to monitor the bionic fruit and enter the picking phase; the recognition phase includes the fruit setting link and the fruit recognition link. The fruit setting link specifically modifies the operating parameters of the bionic fruit through the host computer software or wireless burning software in a wireless or wired manner, such as the cutting time set according to the maturity and fruit type. The force can ensure that the fruit can be picked just right without causing serious damage to the fruit. The fruit color of the target maturity level is selected to prepare for the recognition link and realize the parameter setting of the bionic fruit. The structural settings such as the size setting and weight setting of the simulated fruit can be customized before the experiment begins. The fruit recognition link is specifically for the picking robot system to control the visual recognition module, which is generally an RGBD binocular camera or a monocular image sensor. It approaches and aligns with the data monitoring bionic fruit. The upper epidermis of the data monitoring bionic fruit will show the set mature color. The visual recognition module is deployed with the target detection Yolov8 model. The image processing unit collects and analyzes visual images to obtain maturity status, position information, picking status based on maturity status, probability of successful picking based on maturity status and position information, bionic fruit distribution posture, etc., and then transmits the result parameters required for the picking stage to the robot system control end, so that the robot system can perform the next operation; in the picking stage, when the end effector touches each tactile sensing micro-element 101, when the tactile sensing micro-element 101 produces a pressure resistance change, it is judged that the end effector has been in contact to realize tactile perception, and the end effector continues to grasp until several pressure sensor arrays 302 produce pressure resistance changes. The system generates pressure data to achieve pressure sensing, ultimately separating the permanent magnet 603 from the iron core. The shear force at the moment of separation is measured via a magnetic force meter 702 to achieve shear sensing. The picking phase consists of an approach phase and a picking phase. The approach phase specifically involves the picking robot system executing a picking approach action based on the result parameters transmitted by the vision module. The picking approach action is a preparatory action for the picking phase. During the approach phase, the end effector of the picking robot system needs to move to the preparatory position for the picking phase. However, there is a chance that the end effector and the fruit may collide during the approach phase, causing damage to the fruit. Therefore, touch sensing can effectively record and detect this situation. During the picking phase, the picking robot system controls the end effector to move to the target grasping position and issues a grasping command. The end effector of the system then picks the bionic fruit. Since the end effector of the robot system needs to provide shear force to break the fruit stem to achieve effective picking during the picking process, pressure on the fruit is inevitable during direct grasping. Pressure sensing can effectively detect this situation and record relevant data.In addition, the shearing sensor is distributed at the lower end of the fruit stalk. Before the picking action, the upper and lower ends of the fruit stalk are tightly adsorbed. The threshold of the adsorption magnetic force can be controlled by controlling the excitation coil 701. The magnetic force meter 702 can record the shear force at the moment of cutting. Then it enters the collection stage; during the collection stage, the end effector grabs the picked data-monitoring bionic fruit and moves it to the placement position directly above the fruit basket. Finally, the end effector is released to place the data-monitoring bionic fruit into the fruit basket. The inertial measurement unit IMU module 506 obtains the acceleration data of the data-monitoring bionic fruit in real time during the movement and falling process of the end effector to realize motion perception. The collection stage includes the placement movement link and the collection into the basket link. The placement movement link is specifically the process of the picking robot system moving the bionic fruit from the picking position to the placement position. During this process, the robot movement should be as smooth as possible and not too fast. The acceleration may cause pressure on the fruit, resulting in secondary damage; the motion condition perception of the placement motion link can detect and record the acceleration fluctuations and height changes of the placement motion link; the collection and basketing link mainly refers to the process of the picking robot system placing the bionic fruit from the placement position to the fruit storage device. In this process, if the height is too high when placed in the basket, it may cause secondary damage to the fruit due to falling; the motion condition perception of the collection and basketing link can detect and record the acceleration fluctuations of the placement motion link; the fruit pressure data can be characterized and mapped by the fruit surface pressure data collected by the texture analyzer, and the reasonable pressure range can be determined based on the texture analyzer.

[0036] like Figure 11As shown, the bionic fruit of the present invention is mainly used in the three main stages of fruit picking: the identification stage, the picking stage, and the collection stage. Specifically, it can realize color simulation, touch perception, shear perception, pressure perception, path perception, and fall perception. In specific implementation, the picking process is as follows: data monitors the parameter configuration of the bionic fruit; the picking robot system starts the picking process; data monitors the electrochromic layer 103 of the bionic fruit to simulate the set maturity fruit color; the picking robot system approaches the identification position and enters the identification link; the picking robot system collects image information and analyzes and determines whether the fruit color is mature; the picking robot system obtains the identification result and enters the approach link to pick the bionic fruit of mature color; data monitors the tactile perception microelement 101 of the bionic fruit to detect whether it is touched; the picking robot system approaches the picking position and performs picking ; The pressure sensor array 302 of the bionic fruit records pressure data; the data monitoring magnetic dynamometer 702 of the bionic fruit feeds back shear force; the picking robot system completes picking and enters the collection stage; the data monitoring inertial measurement unit IMU module 506 of the bionic fruit records acceleration changes; the data monitoring inertial measurement unit IMU module 506 of the bionic fruit records falling acceleration; the picking robot system completes the picking process; the captured data includes data from the tactile perception element 101, the pressure sensor array 302, the magnetic dynamometer 702 and the inertial measurement unit IMU module 506.

[0037] like Figure 12 As shown, in this embodiment, a non-structured scene fruit picking method based on bionic fruit data monitoring and driving is specifically provided, which mainly focuses on the above three main stages and six core links, and specifically includes the following steps: S01. Bionic Fruit Parameter Configuration: Before the fruit picking process begins, the bionic fruit parameter configuration link is to modify the operating parameters of the core controller microcontroller unit MCU module 505 of the bionic fruit in a wireless or wired manner through the host computer software or wireless burning software, select and set the fruit color of the target maturity level, prepare for the identification link, and realize the parameter setting of the bionic fruit.

[0038] S02, the picking robot system starts the picking process: The external picking robot system starts the fruit picking process.

[0039] S03. Bionic fruit simulation to set maturity and fruit color: In this link, the bionic fruit mainly controls the electrochromic layer 103 on the upper surface of the epidermal sensing micro-element unit 1 array of the upper epidermal layer to achieve color simulation through the micro-control unit MCU module 505 of the signal and energy supply layer.

[0040] S04. The picking robot system approaches the identification position and enters the identification phase: In this stage, the picking robot system controls the visual recognition module, which is generally an RGBD binocular camera equipped with a Yolov8 recognition model, to approach and aim at the bionic fruit, and the upper epidermis of the bionic fruit will present the set corresponding color.

[0041] S05. The picking robot system collects and analyzes image information: In this stage, the visual recognition module of the picking robot system collects visual images and analyzes them. The specific analysis information includes: maturity status, location information, picking status, probability of picking success, bionic fruit distribution posture, etc. After the analysis is completed, the result parameters required for the picking stage are transmitted to the robot system control end to prepare for the next step.

[0042] S06. The picking robot system obtains the result parameters and enters the approach phase: In this phase, the result parameters transmitted by the vision module of the picking robot system execute the picking approach action. The picking approach action is the preparatory action of the picking phase. In the approach phase, the end effector of the picking robot system needs to move to the preparatory position of the picking phase. However, there is a probability that the end effector will collide with the fruit during the approach phase, causing damage to the fruit. Figure 15 As shown, the present invention simulates the changes in the accelerator curve under different picking stages through the microcontroller unit MCU module 505 and the inertial measurement unit IMU module 506. In this link, the simulation data of the embodiment of the present invention can be used. Figure 15 The fruit contact acceleration signal curve characteristics in (a) are used as a reference to determine whether there is a collision. Figure 15 As shown in (a), this is a simulation of the acceleration change when the robotic arm approaches the fruit. When the simulated fruit is lightly touched, the internal inertial measurement unit IMU module 506 will sense a small mutation in the X-axis and Y-axis acceleration, which can be used to represent an incorrect touch of the approach link.

[0043] S07. The tactile microelement of the bionic fruit detects whether it is touched: In this link, the bionic fruit mainly realizes touch perception by collecting data from the tactile perception micro-element 101 on the lower surface of the epidermal perception micro-element unit 1 array in the upper epidermal layer through the micro-control unit MCU module 505 of the signal and energy supply layer, which can effectively avoid the damage to the fruit caused by the above situation.

[0044] S08. The picking robot system approaches the picking position and performs picking: In this link, the picking robot system controls the end effector to move to the target grasping position and issues a grasping instruction. The end effector of the picking robot system picks the bionic fruit.

[0045] S09. The pressure sensor of the bionic fruit records the pressing data: In this link, the microcontroller unit MCU module 505 of the signal and energy supply layer realizes pressure perception by collecting the pressure data of the pressure sensor array 302 on the force perception frame unit 3 of the lower epidermal layer. During the picking process, in order to achieve effective picking, the end effector of the robot system needs to provide shear force to twist off the fruit stem, so it is inevitable to generate pressure on the fruit when picking by directly grabbing the fruit. Pressure perception can effectively detect this situation and record relevant data. In this link, you can use the following methods: Figure 14 The capacitive flexible sensor in the embodiment of the present invention is used as a pressure sensing method, and the reference data of the embodiment of the present invention is used as a reference data Figure 13 The damage mechanism and mechanical property mapping curve in is used as a reference to establish the relationship between pressure sensor data and damage representation, such as Figure 13 As shown in the figure, the surface pressure of fruits of different maturity and types measured by the texture analyzer can be used as a reference for establishing the mapping relationship between the damage mechanism and mechanical properties during the picking process, such as the pressure measured when pressing the same depth on kiwi, plum, and peach. Figure 14 As shown in the figure, the present invention takes a capacitive flexible sensor as an example, which can collect capacitance data values ​​of fruits of different maturity and types under pressure, and use the data and Figure 13 The texture analyzer values ​​obtained can be used to obtain the mapping relationship between pressure data and electrical values ​​through certain mathematical operations and changes.

[0046] S10, bionic fruit stalk shearing receptor feedback shear force: In this stage, the microcontroller MCU module 505 at the signal and energy supply layer detects shear force by collecting data from the stem shear sensor. Before the fruit is removed, the upper and lower ends of the stem are tightly attached. The microcontroller MCU module 505 controls the magnetic attraction threshold by controlling the excitation coil 701. The magnetic force meter 702 at the lower end records the shear force at the moment of shearing.

[0047] S11, the picking robot system completes picking and enters the collection stage: The collection stage mainly covers the whole process of placing the simulated fruit into the storage device, including the placement movement link and the collection into the basket link. The placement movement link mainly refers to the process of the picking robot system moving the bionic fruit from the picking position to the placement position. During this process, the robot movement should be as smooth as possible. Too fast acceleration may cause pressure on the fruit, resulting in secondary damage; the collection into the basket link mainly refers to the process of the picking robot system placing the bionic fruit from the placement position to the fruit storage device. During this process, if the height is too high when placed in the basket, it may cause secondary damage to the fruit due to falling. In this link, the simulation data of the embodiment of the present invention can be used. Figure 15 (b) and Figure 15The acceleration signal curve characteristics of the fruit placement and basket entry in (c) are used as a reference to determine whether there is a sudden movement of the robot arm and fall damage. Figure 15 As shown in (b), the acceleration change of the robot arm during the placement motion is simulated. When the simulated fruit is placed, a platform-like acceleration change will appear on the X-axis. This platform mutation can be used to represent the sudden motion of the robot arm during the placement motion. Figure 15 As shown in (c), this is a simulation of the acceleration changes in the collection and basket process. When simulating the direct entry of fruits into the basket from a certain height, a large Z-axis acceleration spike will appear at the moment the fruits touch the bottom of the basket. This Z-axis acceleration spike can be used to represent the collision of fruits in the collection and basket process.

[0048] S12. The accelerometer of the bionic fruit records acceleration changes: In this link, the bionic fruit mainly controls the inertial measurement unit IMU module 506 through the microcontroller unit MCU module 505 of the signal and energy supply layer to collect the acceleration change data of the placement path to realize the motion situation perception of the placement movement link.

[0049] S13. The accelerometer of the bionic fruit records the intensity of the fall: In this link, the bionic fruit mainly controls the inertial measurement unit IMU module 506 through the microcontroller unit MCU module 505 of the signal and energy supply layer to collect the acceleration change data of the placement path to realize the perception of the falling intensity in the basket entry link.

[0050] S14. The picking robot system completes the picking process: The external picking robot system ends the fruit picking process.

[0051] In particular, during the above-mentioned picking process, the microcontroller unit MCU module 505 of the signal and energy supply layer controls the communication module 502 to send the collected data to the external data acquisition terminal in real time, thereby realizing data monitoring of the entire fruit picking process.

[0052] This embodiment uses simulated fruit real-time data monitoring to drive fruit picking in unstructured scenarios, realizing dynamic simulation of the entire fruit picking process and multimodal data collection in unstructured scenarios. It can also accurately characterize the damage mechanism and mechanical properties of the picking process based on the collected data, effectively breaking through the bottleneck of data acquisition in traditional picking processes.

[0053] The detailed description of the present invention is only a specific description of the feasible implementation methods of the present invention, and is not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technology of the present invention should be included in the scope of protection of the present invention.

Claims

1. A non-structured scene fruit picking method based on full-process bionic fruit data monitoring, characterized in that: include: Step 1) constructing a data-monitoring bionic fruit with a multi-layer sensing structure, wherein the outer layer of the data-monitoring bionic fruit is used for force sensing and simulating the color of a real fruit; Step 2) The end effector of the picking robot identifies, grasps, and picks the data-monitoring bionic fruit that presents the ripe color of real fruit. At the same time, the data-monitoring bionic fruit obtains grasping data in real time to characterize the physical response characteristics of the data-monitoring bionic fruit during the entire picking process, thereby monitoring the entire picking process of the data-monitoring bionic fruit in real time.

2. The non-structured scene fruit picking method based on full-process bionic fruit data monitoring according to claim 1 is characterized in that: In the step 1), the multi-layer perception structure of the data monitoring bionic fruit includes a plurality of epidermal perception micro-element units (1), an outer structure layer mesh unit (2), a force perception frame unit (3), a cellular structure flesh unit (4) and a signal energy supply structure unit (5) arranged in sequence from the outside to the inside, so as to simulate the upper epidermal layer, the outer structure layer, the lower epidermal layer, the flesh layer and the inner structure layer of the real fruit in sequence. The outer layer structure of the data monitoring bionic fruit includes various epidermal perception micro-element units (1) for contact perception and simulating the color of the real fruit, and is used to support various epidermal perception micro-element units (1). The outer structural layer mesh unit (2) of the micro-element unit (1) and the force sensing frame unit (3) for sensing grasping pressure; the data monitoring bionic fruit also includes a shear sensing unit to simulate the fruit stalk of a real fruit, the top of the signal energy supply structure unit (5) is located at the top center of the shear sensing unit installed on the top of the data monitoring bionic fruit, and the bottom of the shear sensing unit is connected to the top of the signal energy supply structure unit (5); each epidermal sensing micro-element unit (1), the force sensing frame unit (3) and the shear sensing unit are electrically connected to the signal energy supply structure unit (5).

3. The non-structured scene fruit picking method based on full-process bionic fruit data monitoring according to claim 2 is characterized in that: The signal energy supply structure unit (5) includes a communication module (502), an energy supply management module (504), a micro control unit MCU module (505), an inertial measurement unit IMU module (506), an inner structure layer frame (507) and a wireless charging module. The inner structure layer frame (507) is formed as a whole by a hollow spherical main frame and two central axis channels (501). The two central axis channels (501) are vertically arranged at the top center and the bottom center of the spherical main frame respectively. The top of the upper central axis channel (501) is connected to the bottom end of the shear sensing unit. The outer peripheral surface of the spherical main frame is covered with a second substrate layer; the communication module (502), the energy supply management module (504), the micro control unit MCU module (505), the inertial measurement unit IMU module (506) The IMU module (506) and the wireless charging module are both installed inside the spherical main body frame and isolated by a plurality of insulating partition plates (508); the wireless charging module includes a wireless charging coil (509) and a coil base (510), the coil base (510) is installed in the lower part of the inner structural layer frame (507), the wireless charging coil (509) is wound on the coil base (510) and is electrically connected to the energy supply management module (504); the communication module (502), the energy supply management module (504) and the inertial measurement unit (IMU) module (506) are all electrically connected to the micro control unit (MCU) module (505), and the micro control unit (MCU) module (505) is electrically connected to each epidermal perception micro unit (1), the force perception frame unit (3) and the shear perception unit.

4. The non-structured scene fruit picking method based on full-process bionic fruit data monitoring according to claim 3 is characterized in that: The shear sensing unit comprises two fruit stem segments (6, 7), the first fruit stem segment (6) is formed integrally by a ring-shaped hook (601) and a fruit stem (602), the ring-shaped hook (601) is located at the top of the fruit stem (602) to suspend the data monitoring bionic fruit as a whole at a fixed position, a first fruit stem cavity (604) is provided in the bottom of the fruit stem (602) and a permanent magnet (603) is installed therein, a second fruit stem cavity (704) is provided in the interior of the second fruit stem segment (7), an excitation coil (701), an iron core and a magnetic dynamometer (702) are installed in the second fruit stem cavity (704), the excitation coil (701) is wound on the iron core, and the magnetic dynamometer (702) is installed at one end of the iron core, and the excitation coil (701) and the magnetic attraction meter (702) are electrically connected to the micro control unit MCU module (505); when the excitation coil (701) is energized, the end of the permanent magnet (603) is attracted to the other end of the iron core, and at this time, the bottom end of the first stalk segment (6) and the top end of the second stalk segment (7) are connected, and the two stalk cavities (604, 704) are connected; when the bionic fruit is picked, the data monitoring is performed by separating the permanent magnet (603) and the iron core to simulate the fruit stalk cutting process, and the magnetic attraction force when the permanent magnet (603) and the iron core are separated is obtained by the magnetic attraction meter (702) to feedback the shear force threshold at the moment of stalk cutting.

5. The non-structured scene fruit picking method based on full-process bionic fruit data monitoring according to claim 3 is characterized in that: The force perception frame unit (3) comprises a flexible frame (301), a plurality of pressure sensor arrays (302) and a plurality of isolation layers (303). The flexible frame (301) is a hollow spherical structure. The signal energy supply structure unit (5) is located at the inner center of the flexible frame (301). The top and bottom centers of the flexible frame (301) are both provided with fruit stalk interfaces (304) and are respectively rotated on the upper and lower central axis channels (501) of the structural layer frame (507). Each isolation layer (303) is sealed and connected to the flexible frame (301) as the surface of the flexible frame (301) and does not cover the two fruit stalk interfaces (304). Each pressure sensor array (302) is arranged on the outer surface of a respective isolation layer (303). Each pressure sensor array (302) comprises a plurality of pressure sensors arranged at even intervals and is electrically connected to a microcontroller unit MCU module (505). Each pressure sensor array (302) integrates the pressure data of each of its own pressure sensors and transmits the integrated pressure data to the microcontroller unit MCU module (505).

6. The method for picking fruit in a non-structured scene based on full-process bionic fruit data monitoring according to claim 5, characterized in that: The cellular structure pulp unit (4) is composed of a plurality of layers of cellular structure hydrogel arrays, and the cellular structure pulp unit (4) is filled between each isolation layer (303) of the force sensing frame unit (3) and the second substrate layer of the inner structure layer frame (507) of the signal energy supply structure unit (5).

7. The method for picking fruit in a non-structured scene based on full-process bionic fruit data monitoring according to claim 5, characterized in that: The outer structure layer mesh unit (2) is a spherical structure and a mesh envelope structure composed of a plurality of triangles. The flexible frame (301) of the force perception frame unit (3) is inscribed in the inner part of the outer structure layer mesh unit (2) and does not touch each other. The overall frame of the outer structure layer mesh unit (2) adopts a soft tube (201), and the connection position of each triangular frame composed of the soft tube (201) serves as a self-positioning node (202). A microchip is installed on the inner side of each self-positioning node (202) through a plastic sheet, and each microchip has a different position. The skin sensing micro-element unit (1) is installed in each triangular frame and corresponds to a microchip and its position number. Each skin sensing micro-element unit (1) is electrically connected to the microcontroller MCU module (505). The top and bottom centers of the outer structural layer mesh unit (2) form a fruit stalk connection port (203) through the ends of a plurality of soft tubes (201). The two fruit stalk connection ports (203) are respectively connected to the two central axis channels (501) of the structural layer frame (507).

8. The method for picking fruit in a non-structured scene based on full-process bionic fruit data monitoring according to claim 7, characterized in that: Each of the epidermal sensing micro-element units (1) comprises a plurality of tactile sensing micro-elements (101), a first substrate layer (102) and an electrochromic layer (103). For each triangular frame formed by the soft tube (201), the first substrate layer (102) is installed in the triangular frame and serves as the surface layer of the outer structural layer mesh unit (2). The electrochromic layer (103) is attached to the outer side surface of the first substrate layer (102). Each electrochromic layer (103) completely wraps the outer structural layer mesh unit (2). The inner side surface of the first substrate layer (102) serves as an outer surface, and tactile sensing micro-elements (101) are evenly spaced apart. Each tactile sensing micro-element (101) faces the pressure sensor array (302) and does not contact each other. Each tactile sensing micro-element (101) and the electrochromic layer (103) are electrically connected to a microcontroller unit MCU module (505). The microcontroller unit MCU module (505) applies a voltage to the electrochromic layer (103) to change color according to the position number of different microchips.

9. The method for picking fruit in a non-structured scene based on full-process bionic fruit data monitoring according to claim 8, characterized in that: In the step 2), the data monitoring bionic fruit picking process is divided into an identification stage, a picking stage, and a collection stage. In the identification stage, the microcontroller unit MCU module (505) applies a voltage to each electrochromic layer (103) according to the position number of the microchip on the self-positioning node (202) to simulate the ripe color of the real fruit. When the picking robot recognizes that the electrochromic layer (103) presents the ripe color of the real fruit, it controls the end effector to grab the data monitoring bionic fruit and enter the picking stage. In the picking stage, when the end effector touches each tactile sensing microelement (101), when the tactile sensing microelement (101) produces a piezoresistive change, it is determined that the end effector has The end effector continues to grasp until the plurality of pressure sensor arrays (302) generate pressure data to realize pressure perception, and finally separates the permanent magnet (603) and the iron core, obtains the shear force at the separation moment through the magnetic attraction meter (702) to realize shear perception, and then enters the collection phase; in the collection phase, the end effector grasps the picked data monitoring bionic fruit and moves it to the placement position just above the fruit basket, and finally releases the end effector to place the data monitoring bionic fruit into the fruit basket, and the inertial measurement unit IMU module (506) obtains the acceleration data of the data monitoring bionic fruit in the process of moving and falling with the end effector in real time to realize motion perception.

10. The method for picking fruit in a non-structured scene based on full-process bionic fruit data monitoring according to claim 9, characterized in that: Before the identification stage, the data monitoring bionic fruit is first hung via a ring hook (601), and the excitation coil (701) is energized via the microcontroller unit MCU module (505) to apply a preset adsorption force between the permanent magnet (603) and the iron core.

Images