Picking robot carrying multi-modal generation type large model and picking method

Abstract

The invention provides a picking robot carrying a multi-modal generation type large model and a picking method. The picking robot comprises a moving device, a control device, a six-axis mechanical arm, a tail end picking executing mechanism, a tail end camera, a side camera, a global camera and sensors. The control device is loaded with a software program, and the software program is loaded with the applied multi-modal generative large model. According to the picking robot carrying the multi-modal generation type large model and the picking method, the RGB image in the real world is directly mapped into the action of the picking robot, so that the picking robot can learn from visual input and simulate and execute a specific action, and when a task of specific picking operation is executed, the picking operation is more accurate, and the picking efficiency is improved. Data acquired from a field environment are imported into the large model in real time, and according to real-time feedback, the action state is adaptively adjusted in the process of executing picking operation according to a planned path, so that the picking accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of a data big model based on deep learning and an intelligent agricultural robot, and in particular to a picking robot and a picking method equipped with a multimodal generative big model. Background Art

[0002] In the field of modern agriculture, harvesting robots, as an automation technology, have been widely used in orchards and greenhouses to improve harvesting efficiency and reduce labor costs. Traditional harvesting robots mainly rely on the segmented method of perception + planning, that is, first using sensors and visual systems for target detection and positioning, and then using planning algorithms to guide the robot to perform harvesting operations.

[0003] Traditional picking robots and their picking methods face some significant bottlenecks in practical applications. First, traditional perception + planning methods often show low accuracy when dealing with complex and dynamic environments. Due to the different shapes, sizes and colors of fruit trees and fruits, traditional perception systems have difficulty in accurately identifying and classifying fruits, especially under changing lighting and cluttered backgrounds. In addition, the limitations of planning algorithms make it easy for robots to have operational conflicts when handling multiple fruits at the same time, resulting in low picking efficiency and increased fruit damage rate. Second, the net picking rate of existing picking robots in practical applications is often unsatisfactory. This problem mainly stems from the disconnection between perception and planning in the segmented method, which causes the robot to fail to fully consider the actual position and state of the fruit during the picking process, making it impossible to achieve accurate picking operations. In addition, the debugging and optimization of traditional picking robots usually requires a lot of manual intervention, which further increases the implementation cost and complexity. Summary of the invention

[0004] The present invention is proposed in view of the above technical problems, and provides a picking robot and a picking method equipped with a multimodal generative large model. The present invention aims to integrate perception and planning functions by applying an end-to-end multimodal generative large model to the picking robot, and generate picking decisions directly from original multi-source sensor data, thereby eliminating the gap between perception and planning in traditional methods.

[0005] According to a first aspect of an embodiment of the present invention, there is provided a picking robot equipped with a multimodal generative large model, which includes a hardware system and a software program; the hardware system includes a mobile device, a control device, a six-axis robotic arm, an end-point picking actuator, an end-point camera, a side camera, a global camera, and various sensors; the control device is loaded with the software program, and the software program is equipped with the applied multimodal generative large model;

[0006] Wherein, the control device and the six-axis mechanical arm are both installed on the mobile device, and the end picking actuator is installed at the end of the six-axis mechanical arm;

[0007] The method for generating a multimodal generative large model applied to the picking robot includes:

[0008] When the picking robot arrives at the picking site, the operator uses the interactive device to manually control the six-axis mechanical arm and the terminal picking actuator of the picking robot through wireless communication, and simulates the picking action through the interactive device so that the six-axis mechanical arm and the terminal picking actuator of the picking robot can complete various picking tasks in the actual picking environment;

[0009] When the operator controls the six-axis mechanical arm and the terminal picking actuator of the picking robot to perform various picking tasks, the control device automatically records multimodal data, wherein the multimodal data includes the joint angle data of the six-axis mechanical arm, the operation state data of the terminal picking actuator, and the data acquired by each visual camera and each sensor;

[0010] Preprocessing the multimodal data acquired by the control device, wherein the preprocessing includes cleaning processing and enhancement processing; wherein the cleaning processing includes removing noise and incomplete data;

[0011] The preprocessed data is input into the selected deep learning model for training and optimization. During the training process, the loss function is used to evaluate the difference between the model output and the target label, and the model parameters are gradually optimized to obtain the optimal model to form a multimodal generative large model.

[0012] According to a second aspect of an embodiment of the present invention, there is provided a picking robot equipped with a multimodal generative large model as in the first aspect, wherein, during the process of inputting the preprocessed data into a selected deep learning model for training, a normal distribution method is used during data selection to increase the data before and after the picking action, and reduce the data near the target and after the picking is completed.

[0013] According to a third aspect of an embodiment of the present invention, there is provided a picking robot equipped with a multimodal generative large model as in the second aspect, wherein, in the process of acquiring multimodal data, an operator uses an interactive device to simulate picking actions at least twice so that the picking robot can perform at least two picking tasks in an actual environment.

[0014] According to a fourth aspect of an embodiment of the present invention, there is provided a picking method using a picking robot equipped with a multimodal generative large model according to any one of the first to third aspects, the picking method comprising:

[0015] The picking robot arrives at the target picking area, selects a navigation method based on the characteristics of the picking area to navigate to the starting picking point, and then uses the global camera to determine whether there is a fruit picking target. When a fruit picking target appears, the picking robot stops moving;

[0016] After the picking robot reaches the work row at the picking position, it determines whether there is a fruit picking target at the current position within the picking action range of the six-axis robot arm based on the viewing angle of the side camera;

[0017] If it does not exist, the picking robot continues to move forward;

[0018] If it exists, the picking robot controls the six-axis robotic arm to reach the set picking posture, and determines the only picking target according to the set criteria for multiple fruit targets that appear in the camera field of view at the end of the current posture;

[0019] At the same time, the picking robot imports the image data collected by each camera, the data of each sensor, the position information data of the six-axis robotic arm and the terminal picking actuator into the large model applied in the control device in real time. The control device corrects and controls the six-axis robotic arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model to complete the picking operation of the current picking target.

[0020] According to a fifth aspect of an embodiment of the present invention, a picking method as in the fourth aspect is provided, wherein after the picking robot completes the picking operation of the current picking target, the control device uses a large model to control the movement of the six-axis robotic arm and the end picking actuator so as to store the currently picked fruit target into the selected collection box.

[0021] According to a sixth aspect of an embodiment of the present invention, there is provided a picking method as in the fifth aspect, wherein after the picking robot completes the placement of the current picking target, the control device uses the large model to control the six-axis robotic arm to return to the preset observation position and sequentially pick all target fruits in the current field of view according to the set picking order;

[0022] When the picking robot determines that there is no fruit picking target in the current picking area according to the field of view of the end camera and the side camera, the picking robot continues to move until the fruit picking target appears within the motion range of the six-axis robot arm. The picking robot stops moving and repeats the above picking steps to complete the fruit picking in the next area.

[0023] This process is repeated to complete the picking of fruits on one side of the current operation row. When the picking robot reaches the end of the current operation row, it switches the picking direction and completes the picking of fruits on the other side of the current operation row in sequence.

[0024] After the picking robot has finished picking all the fruit targets in the current operation row, it automatically navigates to the next row and repeats the above picking process until all picking operations in the target picking area are completed.

[0025] According to a seventh aspect of an embodiment of the present invention, there is provided a picking method as in the fourth aspect, wherein the control device corrects and controls the six-axis robot arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model, including the generation of a control trajectory, specifically in that the control device generates various control instructions for the joint angles of the six-axis robot arm and the action of the terminal picking actuator according to the real-time data imported into the multimodal generative large model, and transmits the control instructions to the six-axis robot arm and the terminal picking actuator in real time so that the six-axis robot arm and the terminal picking actuator perform the picking action according to the planned path.

[0026] According to an eighth aspect of an embodiment of the present invention, there is provided a picking method as in the seventh aspect, wherein the control device corrects and controls the six-axis robot arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model, and also includes real-time feedback and adaptive adjustment. The specific method is: in the process of performing the specific picking action according to the planned path, the control device continuously monitors the action status of the six-axis robot arm and the terminal picking actuator, and the real-time data continuously fed back by each camera and each sensor is imported into the multimodal generative large model, and the control device issues control instructions in real time to fine-tune and update the motion trajectory and action of the six-axis robot arm and the terminal picking actuator.

[0027] According to a ninth aspect of an embodiment of the present invention, there is provided a picking method as in the fourth aspect, wherein the picking robot imports various data into the large model applied in the control device in real time, and the imported data is specifically:

[0028] {(I1 t , I2 t ), (J1 t , J2 t , J3 t , J4 t , J5 t , J6 t , E t )}, where I1 t is the terminal camera image data at time t, I2 t is the global camera image data at time t, J1 t , J2 t , J3 t , J4 t , J5 t , J6 t is the six-axis joint angle data of the six-axis robot at time t, Et is the state of the terminal picking actuator at time t.

[0029] According to a tenth aspect of an embodiment of the present invention, there is provided a picking method as in the ninth aspect, wherein the state of the terminal picking actuator at time t includes an open state or a closed state, and E t Specifically, it is 0 or 1.

[0030] The beneficial effects of the present invention are as follows: by importing the collected various multi-source sensor data into the deep learning model for training and optimization, applying the end-to-end multimodal generative large model to the picking robot, directly mapping the real-world RGB image to the picking robot action, the picking robot can learn from the visual input and imitate to perform specific actions without the need for additional manually encoded intermediate representations, thereby eliminating the gap between perception and planning of traditional picking robots in the process of performing picking operations, thereby improving not only the picking accuracy of the picking robot, but also the clean picking rate of the picking robot.

[0031] With reference to the following description and the accompanying drawings, specific embodiments of the invention are disclosed in detail, indicating the manner in which the principles of the invention may be employed.

[0032] It should be understood that the embodiments of the present invention are not limited thereby. Features described and / or shown for one embodiment may be used in one or more other embodiments in the same or similar manner, combined with features in other embodiments, or replace features in other embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are included to provide a further understanding of the present invention, and constitute a part of the specification, illustrate preferred embodiments of the present invention, and together with the description are used to explain the principle of the present invention, wherein the same elements are represented by the same reference numerals throughout.

[0034] In the attached picture:

[0035] Figure 1 It is a structural schematic diagram of the harvesting robot equipped with a multi-modal generative large model of the present invention. DETAILED DESCRIPTION

[0036] With reference to the accompanying drawings, the above and other features of the present invention will become apparent through the following description. In the specification and the accompanying drawings, specific embodiments of the present invention are specifically disclosed, which show some embodiments in which the principles of the present invention can be adopted, and it should be understood that the present invention is not limited to the described embodiments.

[0037] The present invention first provides a picking robot equipped with a multi-modal generative large model, the picking robot includes a hardware system and a software program. Figure 1 is a schematic diagram of the structure of the harvesting robot equipped with a multi-modal generative large model of the present invention, see Figure 1 , its hardware system includes a mobile device 1, a control device 2, a six-axis robot arm 3, an end-point picking actuator 4, a terminal camera 5, a side camera (not shown in the figure), a global camera 6 and various sensors (not shown in the figure); wherein, the control device 2 and the six-axis robot arm 3 are both installed on the mobile device 1, and the end-point picking actuator 4 is installed at the end of the six-axis robot arm 3; the terminal camera 5, the side camera, the global camera 6 and various sensors are correspondingly arranged at appropriate positions on the picking robot. Each camera and each sensor is used to obtain environmental and fruit information; according to one embodiment of the present invention, the control device 2 adopts an industrial computer, which serves as a high-performance computing unit for running the trained multimodal generative large model and controlling the actions of each hardware system; the terminal picking actuator 4 can select different picking terminals according to the different fruit objects to be picked. For example, citrus, kiwi and other fruits can use flexible pneumatic picking terminals, apples and other fruits can use claw-type picking terminals, tomatoes, grapes and other fruits can use shearing-type picking terminals. The mechanical properties and sales characteristics of different fruits determine the specific style and picking method of the terminal picking actuator; the chassis of the mobile device 1 can be selected from wheeled chassis, crawler chassis and track chassis according to the different fruit growth environments, so as to meet the needs of different picking scenes.

[0038] According to a preferred embodiment of the present invention, the control device 2 is loaded with a software program, and the software program carries the applied multimodal generative big model. The multimodal generative big model applied to the picking robot needs to collect multi-source sensor data and train the optimization model in advance according to the on-site conditions before the picking robot performs the picking operation, and the finally formed multimodal generative big model is deployed on the picking robot.

[0039] According to a preferred embodiment of the present invention, the method for generating the multimodal generative large model specifically includes:

[0040] S1 picking robot field data collection

[0041] When the picking robot is at the picking site, the operator uses an interactive device [which can be VR glasses, game controllers, mixed reality (MR) devices, etc.] to manually control the action execution of the six-axis mechanical arm 3 and the terminal picking actuator 4 of the picking robot through wireless communication, and simulates the picking action through the interactive device so that the six-axis mechanical arm 3 and the terminal picking actuator 4 of the picking robot can complete various picking tasks in the actual picking environment;

[0042] In the process of the operator controlling the six-axis mechanical arm 3 and the terminal picking actuator 4 of the picking robot to perform various picking tasks, the control device 2 automatically records the joint angle data of the six-axis mechanical arm 3, the operation status data of the terminal picking actuator 4, and the multimodal data such as the data obtained by each visual camera and each sensor. These multimodal data include but are not limited to the shape, color, spatial position, environmental background, etc. of the fruit. Preferably, in the process of obtaining multimodal data, the operator uses the interactive device to simulate the picking action at least twice so that the picking robot performs at least two picking tasks in the actual environment. By simulating the picking tasks multiple times, a large amount of real and diverse operation data is collected, which provides a basis for model training.

[0043] S2 data preprocessing and data enhancement

[0044] The multimodal data acquired by the control device 2 is preprocessed, and the preprocessing includes cleaning and enhancement. After completing the data acquisition, the collected raw data needs to be cleaned to remove noise and incomplete data to ensure the quality of the data set; in order to improve the generalization ability of the model, data enhancement processing can also be performed, such as generating diversified training samples through methods such as image flipping, rotation, and color transformation.

[0045] S3 large model training and optimization

[0046] After completing the data preparation, select a suitable deep learning model architecture, such as a Transformer-based model that can process complex sequence data and adapt to multimodal input. Input the preprocessed data into the selected deep learning model for training and optimization. The model can generate corresponding control strategies by learning the mapping relationship between the actions of the six-axis robot and the characteristics of the fruit. During the training process, the loss function is used to evaluate the difference between the model output and the target label, and the model parameters are gradually optimized to obtain the optimal model to form a multimodal generative large model.

[0047] Preferably, when the preprocessed data is input into the selected deep learning model for training, the normal distribution method is used for data selection to increase the data before and after the picking action and reduce the data near the target and after the picking is completed, so that the model pays more attention to the accuracy of the picking action and improves the success rate of the picking action.

[0048] Correspondingly, the present invention further provides a picking method for performing picking operations using the picking robot equipped with the multimodal generative large model. The picking method of the present invention is specifically as follows:

[0049] The picking robot arrives at the target picking area, selects a navigation method according to the characteristics of the picking area and navigates to the starting picking point, and then uses the global camera 6 to determine whether there is a fruit picking target. When the fruit picking target appears, the picking robot stops moving;

[0050] After the picking robot reaches the work row at the picking position, it determines whether there is a fruit picking target at the current position within the picking action range of the six-axis robot arm 3 according to the viewing angle of the side camera;

[0051] If it does not exist, the picking robot continues to move forward;

[0052] If it exists, the picking robot controls the six-axis robot arm 3 to move to reach the set picking posture, and determines the only picking target according to the set standard for multiple fruit targets appearing in the field of view of the camera 5 at the end of the current posture;

[0053] According to a preferred embodiment of the present invention, the above method for determining the only fruit picking target can be: according to different fruit types, the maturity and picking standards are matched to achieve different varieties at different maturity levels to ensure the best market quality of the fruit, so the picking robot will classify the fruits that appear in the field of view of the terminal camera, select the fruits suitable for picking as the target fruits, and calculate the best picking order according to the distribution of the target fruits in the current field of view; then the first fruit in the best picking order is processed, and the processed image is input as the picking target information into the multimodal generative large model network for picking guidance, so that the large model knows the current only picking target. Of course, in other embodiments, rules such as fruit size and fruit distribution can also be used as standards for judging whether picking is required.

[0054] In determining the above-mentioned optimal picking order, the optimal picking order can be determined according to rules such as from top to bottom, from left to right, etc., or according to the shortest path of the six-axis robot arm 3 movement, the difficulty of picking, etc.

[0055] While determining the unique picking target, the picking robot imports the image data collected by each camera, the data of each sensor, the position information data of the six-axis robot arm 3 and the terminal picking actuator 4 into the large model applied in the control device 2 in real time. The control device 2 corrects and controls the six-axis robot arm 3 and the terminal picking actuator 4 to perform the picking action according to the real-time data imported into the large model to complete the picking operation of the current picking target.

[0056] According to the implementation of the preferred embodiment of the present invention, the control device 2 corrects and controls the six-axis robot 3 and the terminal picking actuator 4 to perform a specific picking process according to the real-time data imported into the large model, including the steps of generating a control trajectory, real-time feedback and adaptive adjustment. Among them, the specific method of generating the control trajectory is: according to the real-time data imported into the multimodal generative large model, the control device 2 generates various control instructions such as the joint angles of the six-axis robot 3 and the actions of the terminal picking actuator 4, and transmits these control instructions to the six-axis robot 3 and the terminal picking actuator 4 in real time so that the six-axis robot 3 and the terminal picking actuator 4 perform the picking action according to the planned path. In addition, in the process of performing a specific picking action according to the planned path, further real-time feedback and adaptive adjustment will be performed. The control device 2 continuously monitors the action state of the six-axis robot 3 and the terminal picking actuator 4, and the real-time data continuously fed back by each camera and each sensor is imported into the multimodal generative large model. The control device 2 issues control instructions in real time to fine-tune and update the motion trajectory and action of the six-axis robot 3 and the terminal picking actuator 4. By performing corrective control during the specific picking process and adapting to new locations and surrounding environments, the picking robot can gradually optimize its control strategy, reduce human intervention, and improve operational efficiency.

[0057] In the process of the picking robot equipped with the multimodal generative large model controlling the six-axis robot arm 3 to perform the picking operation, the state of the terminal picking actuator 4 is also a very important information. According to the preferred embodiment of the present invention, the state of the terminal picking actuator 4 is also imported into the model as data, so the picking robot imports various data into the large model applied in the control device in real time. The imported data is specifically:

[0058] {(I1 t , I2 t ), (J1 t , J2 t , J3 t , J4 t , J5 t , J6 t , E t )}, where I1 t is the image data of the terminal camera 5 at time t, I2 t is the image data of global camera 6 at time t, J1 t , J2 t , J3 t , J4 t , J5 t , J6 t is the six-axis joint angle data of the six-axis robot arm 3 at time t, E tis the state of the terminal picking actuator 4 at time t. In order to simplify the control strategy of the terminal picking actuator 4 and enable it to respond quickly at the picking time, according to a preferred embodiment of the present invention, the input is simplified to an open state and a closed state, that is, the state of the terminal picking actuator 4 at time t includes an open state or a closed state, E t Specifically, it is 0 or 1. The end actuator here can be in various forms such as shearing type, flexible pneumatic type, and clamping type. During specific implementation, the appropriate end actuator type can be selected according to the characteristics of the picking object. Taking the shearing type as an example, state 0 means that the scissors are fully open, and state 1 means that the scissors are fully closed; for the pneumatic type, state 0 means that the negative pressure is closed, and state 1 means that the negative pressure is opened; for the clamping type, state 0 means that the clamp is loose, and state 1 means that the clamp is clamped. This binary state control method is suitable for various types of picking end actuators, which is conducive to the realization of a unified control strategy. At the same time, in the process of data collection and importing the model, it is necessary to ensure that the above data is data at the same time to ensure the accuracy and real-time nature of the input data.

[0059] According to a preferred embodiment of the present invention, after the picking robot completes the picking operation of the current picking target, the control device 2 uses the large model to control the six-axis robot arm 3 and the terminal picking actuator 4 to move so as to store the currently picked fruit target in the selected collection box. The control device 2 not only controls the six-axis robot arm 3 to reach the picking position of the target fruit according to the model, but also controls the terminal picking actuator 4 to pick and grab the fruit. The picking robot determines whether the current picking is completed according to the status of the six-axis robot arm 3 and the terminal picking actuator 4. If it is in the picking completion stage, the main control program of the control device 2 starts to determine whether the fruit is successfully picked. If the fruit is not successfully picked, it will try to pick again; if the fruit is successfully picked (the terminal picking actuator 4 grabs the picking fruit object), the main control program of the control device 2 starts to calculate the placement position of the fruit so as to control the six-axis robot arm 3 to place the picked fruit in the free position of the selected collection box to avoid fruit damage caused by cross-overlapping and stacking of multiple fruits.

[0060] The above method for judging whether picking is completed can be: the picking robot judges what stage the current picking process is in according to the status information of the terminal picking actuator 4. For example, when the terminal picking actuator 4 grabs and picks the fruit for more than 2 seconds, it is judged that the current picking process is in the stage of completion.

[0061] According to a preferred embodiment of the present invention, after the picking robot completes the placement of the current picking target, the control device 2 uses the large model to control the six-axis robot arm 3 to return to the preset observation position and sequentially perform the picking of all target fruits in the current field of view according to the set picking order;

[0062] When the picking robot determines that there is no fruit picking target in the current picking area according to the field of view of the end camera 5 and the side camera, the picking robot continues to move until the fruit picking target appears within the action range of the six-axis robot arm 3, and then stops moving and repeats the above picking steps to complete the fruit picking in the next area;

[0063] The picking of fruits on one side of the current operation row is repeated in this way. When the picking robot reaches the end of the current operation row, it switches the picking direction, including the robot's walking direction, the operation direction of the six-axis robot arm 3, the use of the side camera, etc., and then completes the picking of fruits on the other side of the current operation row in sequence; the picking robot completes the picking of fruits on the other side of the current operation row according to the previous steps and returns to the initial point, thus completing the picking task of the current operation row;

[0064] After the picking robot has finished picking all the fruit targets in the current operation row, it automatically navigates to the next row and repeats the above picking process until all picking operations in the target picking area are completed.

[0065] Through the above steps of the picking method of the present invention, the picking robot can efficiently complete complex picking tasks and ensure the accuracy and adaptability of the operation.

[0066] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and many features and advantages of these embodiments are clear from this detailed description. In addition, since many modifications and changes are easily conceivable to those skilled in the art, it is not intended to limit the embodiments of the present invention to the precise structures and operations illustrated and described, but all suitable modifications and equivalents may be included.

Claims

1. A picking robot equipped with a multi-modal generative large model, characterized in that: The picking robot includes a hardware system and a software program; the hardware system includes a mobile device, a control device, a six-axis robotic arm, an end-point picking actuator, an end-point camera, a side camera, a global camera, and various sensors; the control device is loaded with the software program, and the software program carries the applied multimodal generative large model; Wherein, the control device and the six-axis mechanical arm are both installed on the mobile device, and the end picking actuator is installed at the end of the six-axis mechanical arm; The method for generating a multimodal generative large model applied to the picking robot includes: When the picking robot arrives at the picking site, the operator uses the interactive device to manually control the six-axis mechanical arm and the terminal picking actuator of the picking robot through wireless communication, and simulates the picking action through the interactive device so that the six-axis mechanical arm and the terminal picking actuator of the picking robot can complete various picking tasks in the actual picking environment; When the operator controls the six-axis mechanical arm and the terminal picking actuator of the picking robot to perform various picking tasks, the control device automatically records multimodal data, wherein the multimodal data includes the joint angle data of the six-axis mechanical arm, the operation state data of the terminal picking actuator, and the data acquired by each visual camera and each sensor; Preprocessing the multimodal data acquired by the control device, wherein the preprocessing includes cleaning processing and enhancement processing; wherein the cleaning processing includes removing noise and incomplete data; The preprocessed data is input into the selected deep learning model for training and optimization. During the training process, the loss function is used to evaluate the difference between the model output and the target label, and the model parameters are gradually optimized to obtain the optimal model to form a multimodal generative large model.

2. A harvesting robot equipped with a multi-modal generative large model according to claim 1, characterized in that: In the process of inputting the preprocessed data into the selected deep learning model for training, the normal distribution method is used for data selection to increase the data before and after the picking action and reduce the data near the target and after the picking is completed.

3. A harvesting robot equipped with a multi-modal generative large model according to claim 2, characterized in that: In the process of acquiring multimodal data, the operator simulates picking actions at least twice using an interactive device so that the picking robot performs at least two picking tasks in an actual environment.

4. A harvesting method using a harvesting robot equipped with a multi-modal generative large model as described in any one of claims 1 to 3, characterized in that: The picking method comprises: The picking robot arrives at the target picking area, selects a navigation method based on the characteristics of the picking area to navigate to the starting picking point, and then uses the global camera to determine whether there is a fruit picking target. When a fruit picking target appears, the picking robot stops moving; After the picking robot reaches the work row at the picking position, it determines whether there is a fruit picking target at the current position within the picking action range of the six-axis robot arm based on the viewing angle of the side camera; If it does not exist, the picking robot continues to move forward; If it exists, the picking robot controls the six-axis robotic arm to reach the set picking posture, and determines the only picking target according to the set criteria for multiple fruit targets that appear in the camera field of view at the end of the current posture; At the same time, the picking robot imports the image data collected by each camera, the data of each sensor, the position information data of the six-axis robotic arm and the terminal picking actuator into the large model applied in the control device in real time. The control device corrects and controls the six-axis robotic arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model to complete the picking operation of the current picking target.

5. The picking method according to claim 4, wherein: After the picking robot completes the picking operation for the current picking target, the control device uses the large model to control the action of the six-axis robot arm and the terminal picking actuator so as to store the currently picked fruit target into the selected collection box.

6. The picking method according to claim 5, wherein: After the picking robot completes the placement of the current picking target, the control device uses the large model to control the six-axis robot arm to return to the preset observation position and sequentially pick all target fruits in the current field of view according to the set picking order; When the picking robot determines that there is no fruit picking target in the current picking area according to the field of view of the end camera and the side camera, the picking robot continues to move until the fruit picking target appears within the motion range of the six-axis robot arm. The picking robot stops moving and repeats the above picking steps to complete the fruit picking in the next area. This process is repeated to complete the picking of fruits on one side of the current operation row. When the picking robot reaches the end of the current operation row, it switches the picking direction and completes the picking of fruits on the other side of the current operation row in sequence. After the picking robot has finished picking all the fruit targets in the current operation row, it automatically navigates to the next row and repeats the above picking process until all picking operations in the target picking area are completed.

7. The picking method according to claim 4, wherein: The control device corrects and controls the six-axis robot arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model, including the generation of a control trajectory. The specific method is: according to the real-time data imported into the multimodal generative large model, the control device generates various control instructions for the joint angles of the six-axis robot arm and the action of the terminal picking actuator, and transmits the control instructions to the six-axis robot arm and the terminal picking actuator in real time so that the six-axis robot arm and the terminal picking actuator can perform the picking action according to the planned path.

8. The picking method according to claim 7, wherein: The control device also includes real-time feedback and adaptive adjustment in the process of controlling the six-axis robot arm and the terminal picking actuator to perform the picking action according to the real-time data imported into the large model. The specific method is: in the process of performing the specific picking action according to the planned path, the control device continuously monitors the action status of the six-axis robot arm and the terminal picking actuator, and the real-time data continuously fed back by each camera and each sensor is imported into the multimodal generative large model. The control device issues control instructions in real time to fine-tune and update the motion trajectory and action of the six-axis robot arm and the terminal picking actuator.

9. The picking method according to claim 4, wherein: The picking robot imports various data into the large model applied in the control device in real time, and the imported data are specifically: {(I1 t , I2 t ), (J1 t , J2 t , J3 t , J4 t , J5 t , J6 t , E t )}, where I1 t is the terminal camera image data at time t, I2 t is the global camera image data at time t, J1t ， J2 t , J3 t , J4 t , J5 t , J6 t is the six-axis joint angle data of the six-axis robot at time t, E t is the state of the terminal picking actuator at time t.

10. The picking method according to claim 9, wherein: The state of the end-picking actuator at time t includes open state or closed state, E t Specifically, it is 0 or 1.

Images