Fruit and vegetable picking method based on intelligent mechanical arm

Abstract

The invention discloses a fruit and vegetable picking method based on an intelligent mechanical arm, and belongs to the technical field of fruit and vegetable picking. According to the fruit and vegetable picking description information, a task instruction understanding model based on a large language model is adopted for analysis, and a control signal for controlling a mechanical arm is generated; according to the control signal, a fruit and vegetable intelligent sensing model with intelligence is utilized, the fruit and vegetable category and the grabbing posture are determined through an image collected by a depth camera, and the picking action is adjusted in combination with real-time feedback; according to the control signal, in the process that the mechanical arm executes the picking task, real-time environment perception is conducted through the intelligent technology, and the motion trail is dynamically adjusted through the intelligent obstacle avoidance neural network model so as to avoid the obstacle, according to the fruit and vegetable picking method provided by the invention, a user does not need to manually write a picking instruction code; the operation complexity of the mechanical arm is remarkably reduced, and the accuracy and efficiency of the picking task are improved.

Description

Technical Field

[0001] The invention relates to a fruit and vegetable picking method using a mechanical arm based on embodied intelligence, and belongs to the technical field of fruit and vegetable picking. Background Art

[0002] With the development of robotics technology, robotic arms are becoming more and more important in the field of fruit and vegetable picking. Before performing the picking task, the picking task instructions need to be generated and sent to the robotic arm.

[0003] Existing fruit and vegetable picking technology relies on manually written task instructions and faces problems such as insufficient picking accuracy, rigid operation, and easy damage to fruits and vegetables in complex environments. Robotic arms usually use a single sensor, which makes it difficult to dynamically adjust paths and avoid obstacles, and has limited operational flexibility. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for picking fruits and vegetables using a robotic arm based on embodied intelligence, so that the robotic arm can adaptively adjust the picking action in a complex and changeable environment, thereby solving the problems of insufficient picking accuracy and high operating difficulty in the prior art.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a method for picking fruits and vegetables using a robotic arm based on embodied intelligence, comprising: Acquire fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes feature descriptions and picking requirements of the fruits and vegetables to be picked; According to the fruit and vegetable picking description information, a task instruction understanding model based on a large language model is used to analyze and generate a control signal for controlling the robotic arm; According to the control signal, the fruit and vegetable intelligent perception model of embodied intelligence is used to determine the category and grasping posture of the fruit and vegetable through the images collected by the depth camera, and the picking action is adjusted in combination with real-time feedback; According to the control signal, when the robotic arm performs the picking task, real-time environmental perception is performed through embodied intelligence technology, and the motion trajectory is dynamically adjusted through the intelligent obstacle avoidance neural network model to avoid obstacles.

[0007] Furthermore, the fruit and vegetable picking description information is analyzed by using a task instruction understanding model based on a large language model to generate a control signal for controlling the robotic arm, including: Receiving fruit and vegetable picking description information described in natural language input by a user; Convert the natural language description into multiple word vectors while maintaining contextual relationships; Using self-attention mechanism to identify important information in the description; Based on the pre-trained large language model, combined with the fine-tuning process, control signals are generated to control the robotic arm to perform picking tasks.

[0008] Furthermore, the fruit and vegetable intelligent perception model includes a Transformer layer, a convolutional neural network layer and a fully connected layer.

[0009] Furthermore, the fruit and vegetable intelligent perception model using embodied intelligence determines the category and grasping posture of fruits and vegetables through images captured by a depth camera, including: Performing feature extraction on the RGB image and the depth image through the Transformer layer of the fruit and vegetable intelligent perception model to obtain global visual features of the fruits and vegetables to be picked; Performing feature extraction on the RGB image and the depth image through the convolutional neural network layer of the fruit and vegetable intelligent perception model to obtain local visual features of the fruits and vegetables to be picked; According to the global visual features and the local visual features, the category of the picked fruits and vegetables and the grasping posture are determined through the fully connected layer of the fruit and vegetable intelligent perception model.

[0010] Furthermore, the real-time environmental perception through embodied intelligence technology and the dynamic adjustment of the motion trajectory to avoid obstacles through the intelligent obstacle avoidance neural network model include: Controlling a plurality of depth cameras pre-set around the robotic arm to scan the activity space of the robotic arm at different angles, and obtaining voxel data of the physical environment at different angles around the robotic arm; Through multimodal sensor data fusion, the information collected by the visual sensor, depth sensor, and force sensor from different latitudes is collaboratively processed, and fused with the voxel data of the physical environment at different angles around the robotic arm, and input into the embodied intelligent perception model to extract features of the three-dimensional space. These multimodal features are fused through a weighted mechanism to finally obtain a comprehensive perception result and a three-dimensional space structure; Based on the comprehensive perception results and three-dimensional spatial structure, the motion trajectory is dynamically adjusted to avoid obstacles through the intelligent obstacle avoidance neural network model.

[0011] Furthermore, the architecture of the intelligent obstacle avoidance neural network model includes a three-dimensional convolutional layer, a Transformer layer, a force sensor information processing layer, a multimodal feature fusion layer and a fully connected layer, which are used to dynamically perceive and reconstruct the three-dimensional physical environment around the robotic arm, and combine the force sensor feedback to adjust the motion path and grasping force of the robotic arm in real time.

[0012] In a second aspect, the present invention provides a mechanical arm fruit and vegetable picking device based on embodied intelligence, comprising: A receiving module, used to obtain fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes a feature description and picking requirements of the fruits and vegetables to be picked; A processing module, configured to analyze the fruit and vegetable picking description information using a task instruction understanding model based on a large language model to generate a control signal for controlling the robotic arm; A perception module is used to determine the category and grasping posture of fruits and vegetables based on the control signal and the images captured by the depth camera using the embodied intelligent fruit and vegetable intelligent perception model, and adjust the picking action in combination with real-time feedback; The execution module is used to perform real-time environmental perception through embodied intelligence technology according to the control signal while the robotic arm is performing the picking task, and dynamically adjust the motion trajectory to avoid obstacles through an intelligent obstacle avoidance neural network model.

[0013] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods.

[0014] In a fourth aspect, the present invention provides a computer device, comprising: Memory, for storing computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of any of the aforementioned methods.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of any one of the aforementioned methods when executed by a processor.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides a method for picking fruits and vegetables with a robotic arm based on embodied intelligence. The embodied intelligence technology provides the robotic arm with real-time environmental perception, ensuring dynamic adjustment of picking actions in complex environments. At the same time, it allows flexible adjustment of the gripping force based on force feedback to avoid damage to fruits and vegetables. Embodied intelligence is also combined with reinforcement learning, so that the robotic arm can continuously optimize the picking strategy, improve picking efficiency and accuracy, and reduce the difficulty of operation; the advantages of embodied intelligence greatly improve the environmental adaptability, operational flexibility and picking efficiency of the robotic arm, and solve the limitations of traditional picking technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the application scenario of the fruit and vegetable picking method of a robotic arm based on embodied intelligence provided in this application;

[0019] Figure 2 A schematic diagram of a process flow of an embodiment of a method for picking fruits and vegetables using a robotic arm based on embodied intelligence provided in this application;

[0020] Figure 3 A flowchart of generating a control signal based on a task instruction understanding model provided in this application;

[0021] Figure 4 Schematic diagram of the logical framework of a fruit and vegetable intelligent perception model provided for this application

[0022] Figure 5 A schematic diagram of a process flow of an embodiment of a method for picking fruits and vegetables based on embodied intelligence provided in this application;

[0023] Figure 6 A schematic diagram of the logical framework of an intelligent obstacle avoidance neural network model provided in this application. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0025] Embodiment 1: This embodiment introduces a method for picking fruits and vegetables using a robotic arm based on embodied intelligence, comprising: Acquire fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes feature descriptions and picking requirements of the fruits and vegetables to be picked; According to the fruit and vegetable picking description information, a task instruction understanding model based on a large language model is used to analyze and generate a control signal for controlling the robotic arm; According to the control signal, the fruit and vegetable intelligent perception model of embodied intelligence is used to determine the category and grasping posture of the fruit and vegetable through the images collected by the depth camera, and the picking action is adjusted in combination with real-time feedback; According to the control signal, when the robotic arm performs the picking task, real-time environmental perception is performed through embodied intelligence technology, and the motion trajectory is dynamically adjusted through the intelligent obstacle avoidance neural network model to avoid obstacles.

[0026] The application process of the fruit and vegetable picking method based on embodied intelligence provided by the present embodiment specifically involves the following steps: First, the terms involved in this application are explained: Embodied intelligence: refers to an intelligent system that can interact with the environment in real time through perception. The embodied intelligent system integrates the capabilities of perception, cognition, decision-making and action. It can perceive the environment through sensors, complete tasks through actuators, and make adaptive decisions and adjustments according to changes in the environment to ensure operational flexibility and accuracy in complex dynamic environments.

[0027] Big language model: refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text; Multimodal sensor: refers to the acquisition of environmental and object information from different dimensions by integrating multiple types of sensors (such as visual sensors, depth sensors, force sensors, etc.). Multimodal sensor data provides more comprehensive environmental perception after fusion processing, ensuring that the system can perform high-precision tasks in complex environments.

[0028] At present, with the development of smart agriculture and automation technology, robotic arms are gradually being applied to the field of fruit and vegetable picking. Among them, the automated fruit and vegetable picking technology based on robotic arms has become a research hotspot. With the assistance of robotic arms, fruit and vegetable picking tasks can be highly automated. However, before applying robotic arms to implement fruit and vegetable picking tasks, it is usually necessary to generate corresponding picking task instructions in advance and send these instructions to the robotic arms to ensure that the picking tasks can be accurately executed.

[0029] At present, the picking task instructions sent to the robotic arm mainly rely on manually written codes. Specifically, based on the actual picking needs of fruits and vegetables, the user manually writes the instruction code of the picking task, and sends it to the robotic arm after writing, so as to realize the real-time control of the robotic arm and complete the fruit and vegetable picking task. At the same time, in the real-time control process of the robotic arm, the industrial communication bus or robot operating system is usually used to communicate with the robotic arm.

[0030] However, in the existing task instruction generation scheme, users need to have high programming skills, so the control of the robot arm is difficult. In addition, the posture of fruits and vegetables in different growth environments and positions may vary greatly. The current picking method usually relies on the robot arm to perform the picking operation after the fruits and vegetables are in a specific position. When the posture or growth environment of fruits and vegetables changes, the picking efficiency and accuracy of the robot arm will be greatly affected, and the existing picking method lacks adaptability to diversified scenarios, which is not conducive to the flexible development and promotion of smart agricultural production lines. In addition, the existing communication control method is relatively complex and has a high learning cost, which further increases the difficulty of controlling the robot arm. At the same time, when the robot arm is active in complex environments such as orchards, if there is no effective obstacle avoidance solution, it may collide with surrounding obstacles or fruit trees, causing safety hazards. Therefore, how to design a vision-based dynamic obstacle avoidance method that enables the robot arm to perceive the surrounding three-dimensional environment in real time and avoid collisions, so as to safely and efficiently perform fruit and vegetable picking tasks, has become a technical problem that needs to be solved urgently.

[0031] In response to the above problems, the inventors discovered during their research on the method of using a robotic arm to pick fruits and vegetables that the key to realizing automated fruit and vegetable picking with a robotic arm is to be able to generate and transmit picking task instructions to the robotic arm. However, the current method of generating picking task instructions relies on manually written code, and users need to have high programming skills. In addition, when the posture of fruits and vegetables changes, it is difficult for the robotic arm to accurately identify and perform picking tasks. Based on this, the inventors considered whether it is possible to design a network model that automatically generates task instructions through natural language descriptions, thereby getting rid of the limitations of manually written code. Based on this, the inventors proposed a task instruction understanding model, which takes the fruit and vegetable picking description information input by the user as input, generates a control signal that controls the robotic arm to perform picking tasks, and realizes the automatic generation of picking instructions.

[0032] On this basis, the inventor designed a fruit and vegetable intelligent perception model based on embodied intelligence. The model uses multimodal sensor fusion to take the collected fruit and vegetable images and force feedback information as input, and outputs the category and grasping posture of fruits and vegetables, thereby realizing accurate identification and picking of fruits and vegetables in different postures. At the same time, in the process of the robotic arm performing the picking task, combined with embodied intelligence technology, the environment around the robotic arm is perceived in real time, and the motion trajectory of the robotic arm is dynamically adjusted to ensure the safety and efficiency of the picking task.

[0033] Figure 1 Schematic diagram of the application scenario of the fruit and vegetable picking method based on embodied intelligence provided in this application. Figure 1 As shown, the application scenario of the present invention includes a terminal device 100, a server 101, a depth camera 102, a force sensor 103 and a robotic arm 104. The server 101 is used to deploy a task instruction understanding model 1011, a fruit and vegetable intelligent perception model 1012 and an intelligent obstacle avoidance neural network model 1013.

[0034] The terminal device 100 receives the fruit and vegetable picking description information input by the user and transmits the information to the server 101. After receiving the picking description information sent by the terminal device 100, the server 101 inputs it into the task instruction understanding model 1011 for analysis and processing, and generates a signal to control the robotic arm 104 to perform the picking task. The control signal will be transmitted to the perception system of the robotic arm 104 in real time, and the robotic arm 104 will perform the fruit and vegetable picking operation according to the signal.

[0035] Before the robot arm 104 performs the picking operation, the depth camera 102 will take images of the fruits and vegetables to be picked on the fruit tree as input to the fruit and vegetable intelligent perception model 1012 in the server 101 to determine the category and grasping posture of the fruits and vegetables. In addition, the depth camera 102 will also monitor the surrounding environment of the robot arm 104 in real time, and input the environmental data into the intelligent obstacle avoidance neural network model 1013 of the server 101 to plan the motion trajectory of the robot arm, and at the same time collect the force feedback information obtained by the force sensor 103, and input the force feedback data into the intelligent obstacle avoidance neural network model 1013 of the server 101 to adjust the robot arm to avoid obstacles in real time. Based on the control signals generated by the task instruction understanding model, the category and grasping posture of fruits and vegetables, and the motion trajectory provided by the intelligent obstacle avoidance model, the robot arm 104 performs precise fruit and vegetable picking operations.

[0036] although Figure 1 Only one terminal device 100, one server 101, one industrial camera 102, one force sensor 103 and one robotic arm 104 are shown, but in actual applications, there may be multiple terminal devices, servers, industrial cameras and robotic arms.

[0037] The technical solution of the present invention will be described in detail below in conjunction with specific embodiments.

[0038] Figure 2 This is a flow chart of the first embodiment of the method for picking fruits and vegetables with a robotic arm based on embodied intelligence provided in this application.

[0039] like Figure 2 As shown, the process of the fruit and vegetable picking method based on embodied intelligence by a robotic arm may include: S201: receiving fruit and vegetable picking description information input by a user, wherein the fruit and vegetable picking description information includes a feature description of the fruit and vegetables to be picked and a picking requirement description.

[0040] In this step, when the user has a fruit and vegetable picking request, the user can input the pre-set picking description information in the terminal device, and the information is transmitted to the server through the terminal device. The server is mainly used to receive the fruit and vegetable picking description information input by the user. Specifically, the fruit and vegetable picking description information includes a feature description of the fruit and vegetables to be picked and a description of the picking requirements, which are usually composed of natural language descriptions. For example, the user may enter "pick all ripe apples" or "pick apples larger than 10 cm in diameter."

[0041] Feature description mainly refers to the limitation of the attributes of fruits and vegetables to be picked, such as the type, size, maturity, etc. The picking requirement description mainly refers to the specific instructions for picking requirements, such as whether to sort by size, whether to pick only fruits of a certain maturity, etc.

[0042] S202: According to the fruit and vegetable picking description information, the task instruction understanding model is used to analyze and process, and a control signal for controlling the robot arm to pick the fruit and vegetables to be picked is obtained. The task instruction understanding model is a neural network model pre-trained based on a large language model, which extracts picking requirements from the fruit and vegetable picking description information and generates picking control signals based on the requirements.

[0043] In this step, based on step S201, the task instruction understanding model deployed in the server performs analysis and processing based on the description information after receiving the fruit and vegetable picking description information input by the user. The task instruction understanding model understands the input natural language through the large language model and generates a control signal for controlling the robot arm to perform the picking task.

[0044] Specifically, the task instruction understanding model is processed based on the pre-training results of the Large Language Model (LLM). The large language model can effectively extract the characteristics of fruits and vegetables and picking requirements input by users by learning from a large amount of text data. During the pre-training stage, the large language model uses a variety of technical means, such as negative example sampling and word blocking strategies, to gradually improve the model's ability to understand the semantics and syntactic structure of the text. In this process, the model is optimized through massive language data, giving it strong semantic understanding and reasoning capabilities.

[0045] When a user enters a natural language description, such as "pick all the ripe red apples", the large language model first converts the natural language into multiple word vectors while maintaining the relationship between the contexts, so that the model can understand the association between words and the intention of the entire text. Next, the model uses the self-attention mechanism to identify important information in the description, ensuring that the model can accurately grasp the dependencies between words and sentences, and give higher attention weights to key information in complex scenarios. This mechanism enables the model to understand multiple conditions (such as maturity, color, size) and generate corresponding instructions when processing descriptions of fruit and vegetable picking.

[0046] After the large language model is pre-trained, task-related fine-tuning is required to adapt to the task of generating fruit and vegetable picking control signals. Through fine-tuning, the model can respond to input more accurately when processing specific fruit and vegetable picking tasks. Specifically, the fine-tuning process includes adjusting the model structure so that it can generate specific robot arm control signals based on the characteristics and needs of fruits and vegetables. The generated picking control signal is a set of instruction code sequences that instruct the robot arm to perform subsequent picking operations.

[0047] The fine-tuning process mainly adjusts the last few layers of the large language model to adapt to tasks with different fruit and vegetable types and picking requirements. This process adjusts the model's hierarchical structure so that it can accurately classify the characteristics of different fruits and vegetables (such as type, size, and maturity), and converts this information into operating instructions for the robotic arm to ensure accurate execution of the picking task.

[0048] In one implementation, the classification network consists of three fully connected layers and one softmax classification layer. The first fully connected layer contains 4096 nodes for processing high-dimensional features, the second fully connected layer contains 1024 nodes to further streamline the feature space, and the third fully connected layer contains 256 nodes for generating compressed feature representations. The softmax classification layer is used to classify the input natural language description into categories, such as "picking by maturity" or "picking by size" and other task categories. The specific number of output nodes depends on the requirements of the picking task.

[0049] For example, when the user inputs "pick all ripe apples" or "pick all fruits with a diameter greater than 10 cm", the task instruction understanding model will divide these description information into two task dimensions in the classification network: the first dimension is "pick by fruit and vegetable type" or "pick by fruit and vegetable maturity", and the second dimension is "pick by size". The model will combine these two classification tasks to generate a picking control signal for the robot arm to execute, that is, a sequence of operation instruction codes for the robot arm. The control signal will be transmitted to the perception system of the robot arm so that the robot arm can perform specific fruit and vegetable picking operations.

[0050] In one possible implementation, the loss function in the task instruction understanding model is defined as follows: ;

[0051] in is the function value, Indicates Fruits and vegetables, Indicates the first categories, is the total number of types of fruits and vegetables to be picked, is the total number of categories in the task dimension is the actual class label, is the predicted probability of the model output. Through the minimum loss function, the model can continuously optimize the classification accuracy of the picking task and ensure that the generated control signal can effectively guide the robot arm to complete the complex fruit and vegetable picking task.

[0052] Figure 3The present application provides a flow chart for generating a control signal based on a task instruction understanding model. Figure 3 As shown, the task instruction understanding model receives the natural language input by the user, which is the description information of the fruits and vegetables to be picked. The task instruction understanding model is fine-tuned based on the pre-trained large language model. Based on the processing of the natural language input by the task instruction understanding model, a control signal for controlling the robot arm to perform the fruit and vegetable picking task can be obtained. The control signal is transmitted to the perception system of the robot arm in real time to control the robot arm to perform the parts sorting task.

[0053] S203: According to the control signal, the fruit and vegetable intelligent perception model is used to collect images and data of fruits and vegetables through multimodal sensors (such as RGB cameras, depth cameras, etc.), and the categories and grasping postures of the fruits and vegetables to be picked are determined.

[0054] In this step, based on step S202, the task instruction understanding model deployed in the server completes the analysis and processing of the fruit and vegetable picking description information and generates a control signal for controlling the robotic arm to perform the fruit and vegetable picking task, and then inputs the control signal into the perception system of the robotic arm.

[0055] When the perception system of the robotic arm receives the control signal, it starts to perform the fruit and vegetable picking task. During the picking process, multimodal sensors (such as RGB cameras, depth cameras, and force sensors) capture and feedback images of each fruit and vegetable to be picked in real time, and transmit the captured image data to the fruit and vegetable intelligent perception model deployed in the server in real time. The fruit and vegetable intelligent perception model analyzes and processes the captured fruit and vegetable images to determine the category of fruits and vegetables and the corresponding grasping posture, thereby assisting the robotic arm to achieve accurate recognition and picking operations of fruits and vegetables.

[0056] Among them, the fruit and vegetable intelligent perception model is a pre-trained neural network model, which is specifically used to determine the category and grasping posture of fruits and vegetables based on image recognition and analysis. In the fruit and vegetable intelligent perception model, the input is the image data of the fruits and vegetables to be picked, and the output is the category of the fruits and vegetables and the appropriate grasping posture. Through the analysis of image features, the model can accurately identify the types of fruits and vegetables, and further calculate the optimal grasping method based on their position and posture. Based on the determined category of fruits and vegetables, the robotic arm combines the picking instructions to determine whether to pick the fruits and vegetables. If the category of the fruits and vegetables meets the specified category in the picking instructions, the robotic arm will pick the fruits and vegetables from the fruit tree and place them at the target location based on the grasping posture generated by the model. If the category of the fruits and vegetables does not match the picking instructions, the robotic arm will not operate on the fruits and vegetables.

[0057] Specifically, the fruit and vegetable intelligent perception model extracts global and local features from the image through multi-layer convolutional neural networks (CNNs). After multi-layer feature fusion, it finally determines the category C and grasping posture P of the fruit and vegetable. Assuming that the input image of the model is I, the model obtains the feature vector f(I) through a series of convolution operations and pooling operations, and then classifies it through the fully connected layer to obtain the category prediction: ;

[0058] in, and are the weights and biases of the classification layer, respectively. is the feature extracted from the image, and C is the probability distribution of the fruit and vegetable categories. The grasping posture P is obtained through the regression network, and the model optimizes the accuracy of the grasping posture by minimizing the mean squared error (MSE) between the predicted grasping posture and the actual posture: ;

[0059] in, is the grasping posture of the i-th fruit and vegetable predicted by the model, is the actual grasping posture, and N is the total number of fruits and vegetables. Through the optimization of this loss function, the robotic arm can accurately perform the grasping operation on fruits and vegetables.

[0060] During the picking process, the gripping posture determined by the robot arm based on the intelligent perception model of fruits and vegetables can effectively cope with the changes in the positions and angles of fruits and vegetables, thereby achieving accurate and stable gripping. If the category of fruits and vegetables meets the picking instructions, the robot arm will pick the fruits and vegetables based on the gripping posture and place them at the specified target location; otherwise, the robot arm will skip the fruits and vegetables and not perform any operation. The determination of the gripping posture is crucial to coping with complex fruit tree environments. It can ensure that the robot arm will not mistakenly grasp or damage the fruits when handling multiple fruits and vegetables.

[0061] S204: When the robot arm is performing the picking task, it uses embodied intelligence technology to perform real-time environmental perception, and dynamically adjusts the motion trajectory to avoid obstacles through the intelligent obstacle avoidance neural network model. The robot arm senses obstacles such as branches and leaves in the environment through force sensors, and dynamically adjusts the motion trajectory of the robot arm in combination with force feedback to ensure collision avoidance during the picking process and ensure the safety and efficiency of picking.

[0062] In this step, based on the control signal generated in step S202 to control the robot arm to perform fruit and vegetable picking, and the category and grasping posture of the fruit and vegetable to be picked obtained in step S203, the robot arm will start to perform the picking task. At the same time, through the intelligent obstacle avoidance neural network model and the built-in force sensor of the robot arm, real-time dynamic obstacle avoidance is achieved during the picking process, ensuring that the robot arm can effectively avoid obstacles in complex environments.

[0063] The robotic arm is equipped with a built-in perception system, which includes multimodal sensors (such as RGB cameras, depth cameras) and force sensors, and is equipped with an embedded artificial intelligence model. The visual system captures the image data of the fruits and vegetables to be picked in real time through sensors, and transmits the images to the intelligent perception model of fruits and vegetables. The intelligent perception model analyzes the category and grasping posture of fruits and vegetables based on image information, and adjusts the grasping force and angle of the robotic arm in combination with the feedback information of the force sensor to ensure that the fruits and vegetables will not be damaged due to improper grasping. At the same time, the intelligent obstacle avoidance neural network model analyzes the movement path of the robotic arm in real time based on the environmental information obtained by the vision and depth sensors, and combines the physical feedback of the force sensor to ensure that the movement trajectory of the robotic arm is dynamically adjusted before it contacts obstacles.

[0064] The force sensor can detect in real time the physical feedback received by the robot arm during the picking process, such as collisions, resistance or force changes between the robot arm and obstacles. When the robot arm approaches an obstacle (such as a tree branch), the force sensor will sense the change in external force and transmit this information to the obstacle avoidance system. Combined with the visual and depth information of the intelligent obstacle avoidance neural network model, the system can dynamically adjust the path of the robot arm to prevent collisions. For example, when the robot arm touches the branches of a fruit tree, the force sensor will detect the pressure from the branches and trigger the robot arm to adjust its action to avoid damaging the structure of the fruit tree while continuing to efficiently pick the fruit.

[0065] When the robotic arm performs a picking task, the control signal, fruit and vegetable category, grasping posture, and feedback from the force sensor work together to determine whether the current fruits and vegetables meet the picking requirements. For example, if the control signal requires "picking all ripe red apples", and the currently detected fruit and vegetable category does not meet the requirements, the robotic arm will not perform the picking operation; if it meets the requirements, the robotic arm will optimize the grasping action based on the obstacle avoidance path generated by the intelligent obstacle avoidance neural network model and the real-time feedback from the force sensor to ensure that the robotic arm can accurately grasp fruits and vegetables in complex environments and place them in the specified position. Through the physical feedback provided by the force sensor, the robotic arm can adjust the grasping force in real time to ensure that the fruits and vegetables are not damaged.

[0066] The fruit and vegetable picking method based on embodied intelligence provided in this embodiment describes in detail how the robotic arm accurately performs the entire process of the picking task based on control signals, fruit and vegetable categories, grasping postures, and feedback from force sensors. The task instruction understanding model can automatically generate instructions for controlling the robotic arm to perform picking based on the picking description information input by the user, reducing the complexity of the operation. The fruit and vegetable intelligent perception model can accurately identify the types of fruits and vegetables and determine the grasping postures, avoiding the requirement that fruits and vegetables must appear in a specific posture before picking. The combination of the intelligent obstacle avoidance neural network model and the force sensor ensures the dynamic obstacle avoidance processing of the robotic arm when performing picking tasks, avoids collisions between the robotic arm and obstacles, and ensures the safety and efficiency of the task.

[0067] The combination of the intelligent obstacle avoidance system and the force sensor is reflected in the optimization of the robot's obstacle avoidance path. The current motion trajectory of the robot is set to T(t), the path of the obstacle is O(t), and the external force change detected by the force sensor is F(t). The obstacle avoidance goal can be achieved by minimizing the following comprehensive loss function: ;

[0068] in, is the distance between the robot arm and the obstacle at time t, is a constant that prevents the denominator from being zero, is the magnitude of the external force detected by the force sensor, is the coefficient that weighs visual obstacle avoidance and force feedback. By minimizing this loss function, the robot arm can avoid obstacles while using force feedback to make dynamic adjustments to ensure safe and stable operations.

[0069] In a possible implementation, based on the above embodiment, Figure 4 This is a logical framework diagram of a fruit and vegetable intelligent perception model provided in this application. Figure 4 As shown in the figure, the fruit and vegetable intelligent perception model takes the RGB image and depth image of the fruits and vegetables to be picked as input. First, the Transformer layer receives the input RGB image and depth image, and after processing, transmits the processed data to the convolutional neural network layer. After processing by the convolutional neural network layer, the visual features of the fruits and vegetables are extracted, and the processing results are then transmitted to the fully connected layer. Under the processing of the fully connected layer, the category and grasping posture of the fruits and vegetables are finally output as the output of the fruit and vegetable intelligent perception model.

[0070] Figure 5 This is a flow chart of an embodiment of a method for picking fruits and vegetables based on embodied intelligence provided in this application. Figure 5As shown, based on the above embodiment, an intelligent obstacle avoidance neural network model is used to process the voxel data of the physical environment at different angles around the robotic arm collected in real time by multimodal sensors (such as RGB cameras and depth cameras) to obtain the three-dimensional spatial structure around the robotic arm. The process of the fruit and vegetable picking method based on embodied intelligence may include:

[0071] S1001: Control multiple cameras pre-set around the robotic arm to scan the activity space of the robotic arm at different angles to obtain voxel data of the physical environment at different angles around the robotic arm.

[0072] In this step, when the robotic arm picks fruits and vegetables according to the control signal, the type of fruits and vegetables to be picked, and the grasping posture, an intelligent obstacle avoidance neural network model is needed to generate a three-dimensional spatial structure around the robotic arm to ensure the safety of the picking process.

[0073] The input of the intelligent obstacle avoidance neural network model is the voxel data of the physical environment around the robot arm at different angles. These data are collected in real time by multiple cameras pre-set around the robot arm. Specifically, multiple cameras are pre-set around the robot arm. When the robot arm performs the fruit and vegetable picking task, the camera will scan the working area of ​​the robot arm in real time to obtain the environmental voxel data at different angles.

[0074] S1002: Input the voxel data of the physical environment at different angles around the robotic arm into the intelligent obstacle avoidance neural network model, extract and reconstruct three-dimensional spatial features, and generate a three-dimensional spatial structure around the robotic arm.

[0075] In this step, based on the voxel data of the physical environment at different angles around the robotic arm obtained in step S1001, all data are input into the intelligent obstacle avoidance neural network model in real time. The model will extract three-dimensional spatial features and reconstruct the environment around the robotic arm in three dimensions to generate a three-dimensional spatial structure of the environment.

[0076] Among them, the three-dimensional spatial structure reconstructed by the intelligent obstacle avoidance neural network model will provide spatial motion constraints for the robot arm to perform fruit and vegetable picking tasks. Based on these spatial constraints and combined with the kinematic model of the robot arm, the intelligent obstacle avoidance neural network model can plan a reasonable and safe robot arm motion trajectory, thereby ensuring that the robot arm will not collide or have safety issues when performing fruit and vegetable picking tasks.

[0077] The fruit and vegetable picking method based on embodied intelligence provided in this embodiment mainly describes how the intelligent obstacle avoidance neural network model obtains the three-dimensional spatial structure of the physical environment around the robot arm. The intelligent obstacle avoidance neural network model uses voxel data collected by multi-angle sensors (such as depth cameras and RGB cameras) as input, generates a three-dimensional spatial structure after processing, and then provides the necessary constraints for the motion trajectory planning of the robot arm. Based on this information, the system can plan a safe motion path for the robot arm in a complex environment, thereby ensuring the safety and efficiency of the robot arm in performing fruit and vegetable picking tasks in an orchard environment.

[0078] In one possible implementation, the architecture of the intelligent obstacle avoidance neural network model includes a three-dimensional convolutional layer, a Transformer layer, a force sensor information processing layer, a multimodal feature fusion layer, and a fully connected layer. When the robot arm performs fruit and vegetable picking tasks, the model can dynamically perceive and reconstruct the surrounding three-dimensional physical environment, and combine the force sensor feedback to adjust the robot arm's motion path and grasping force in real time, thereby ensuring that the robot arm can complete fruit and vegetable picking safely and efficiently in a complex environment.

[0079] Specifically, the intelligent obstacle avoidance neural network model first extracts the global and local visual features of the three-dimensional space from the voxel data at different angles around the robot arm through a three-dimensional convolution layer. The function of this layer is to extract the three-dimensional structure of obstacles and fruit trees from the spatial data collected by multimodal sensors (such as depth cameras and RGB cameras). Through convolution operations, this layer can capture the depth information, shape, and relative position of obstacles in the environment.

[0080] Subsequently, the Transformer layer receives the convolutional data, analyzes the time series changes during the robot's movement, and captures the dynamic changes of the environment. Through the self-attention mechanism, this layer can effectively identify and predict the movement trends of obstacles around the robot during its movement, provide support for dynamic obstacle avoidance, and ensure that the robot can flexibly respond to changes in the environment during its movement.

[0081] When the robot arm comes into contact with a fruit tree or other obstacle, the force sensor information processing layer will process the feedback information from the force sensor and sense the size and direction of the robot arm's touch force in real time. This layer can not only detect whether the robot arm is in contact with an obstacle, but also evaluate the robot arm's grasping force on fruits and vegetables, ensuring that the robot arm can automatically adjust the grasping force based on the contact force feedback to avoid damage to fruits and vegetables or insufficient grasping force.

[0082] Next, the model integrates information from different sensors (vision, depth, force) through a multimodal feature fusion layer. The main function of this layer is to fuse multimodal data so that the model can comprehensively consider environmental data from multiple dimensions. This layer combines visual information, depth information, and force feedback to generate a more accurate environmental model, ensuring that the robotic arm can operate accurately in the complex and changing orchard environment.

[0083] Finally, the fully connected layer processes the fused multimodal features and generates the final control signal. The output of the fully connected layer not only includes the obstacle avoidance path planning results of the robot arm, but also reconstructs the three-dimensional space model based on environmental data and force feedback, and updates the motion trajectory of the robot arm in real time. By controlling the grasping force, the robot arm can dynamically adjust the grasping strategy to ensure that fruits and vegetables are safely picked in different positions and conditions.

[0084] Figure 6 A schematic diagram of the logical framework of an intelligent obstacle avoidance neural network model provided in this application. Figure 6 As shown in the figure, during the fruit and vegetable picking process, the robot arm reconstructs the 3D physical environment around the robot arm through multimodal feature fusion technology based on vision, depth and force feedback. Through this reconstruction, the model can mark the non-collisionable area around the robot arm in real time, and perform dynamic obstacle avoidance and motion trajectory planning on this basis to ensure that the robot arm can complete the fruit and vegetable picking task safely and stably.

[0085] The intelligent obstacle avoidance neural network model uses deep learning technology to process voxel data and force feedback information collected by multimodal sensors. First, the model uses a three-dimensional convolutional layer to extract global and local features of the surrounding environment, and uses a Transformer layer to process dynamic changes during the movement of the robotic arm. Next, the force sensor information processing layer analyzes the contact between the robotic arm and fruits, vegetables, and obstacles, and adjusts the movement and grasping force of the robotic arm in real time. The multimodal feature fusion layer further fuses visual, depth, and force information to generate the final control signal, and reconstructs a three-dimensional spatial model of the same size as the input voxel grid through a fully connected layer.

[0086] Based on the intelligent obstacle avoidance neural network model and the kinematic model of the robotic arm, the system can generate the motion trajectory of the robotic arm, ensuring that it can efficiently perform fruit and vegetable picking tasks in complex orchard environments. In this process, the real-time feedback provided by the force sensor not only helps the robotic arm avoid collisions with obstacles, but also ensures the reasonable control of the gripping force during fruit and vegetable picking, avoiding damage to the fruits and vegetables or affecting the picking efficiency.

[0087] In one implementation, the loss function of the intelligent obstacle avoidance neural network model during pre-training is defined as follows: ;

[0088] in, and is the adjustment coefficient, satisfying . Measures the difference between the reconstructed three-dimensional spatial structure and the actual environment. The formula is: ;

[0089] Where N is the number of voxels, is the actual voxel label, It is the predicted voxel value output by the intelligent obstacle avoidance neural network model. is a regularization loss term used to make the reconstructed three-dimensional structure smoother. The formula is: ;

[0090] By optimizing this loss function, the intelligent obstacle avoidance neural network model can accurately reconstruct the three-dimensional environment around the robotic arm and provide dynamic obstacle avoidance support for the robotic arm.

[0091] Through the collaborative work of the three-dimensional convolutional layer, average pooling layer and fully connected layer of the intelligent obstacle avoidance neural network model, combined with real-time input environmental data, the model can plan a safe motion trajectory for the robotic arm, ensuring that obstacles are effectively avoided in fruit and vegetable picking tasks while completing precise grasping operations.

[0092] Embodiment 2: This embodiment provides a mechanical arm fruit and vegetable picking device based on embodied intelligence, comprising: A receiving module, used to obtain fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes a feature description and picking requirements of the fruits and vegetables to be picked; A processing module, configured to analyze the fruit and vegetable picking description information using a task instruction understanding model based on a large language model to generate a control signal for controlling the robotic arm; A perception module is used to determine the category and grasping posture of fruits and vegetables based on the control signal and the images captured by the depth camera using the embodied intelligent fruit and vegetable intelligent perception model, and adjust the picking action in combination with real-time feedback; The execution module is used to perform real-time environmental perception through embodied intelligence technology according to the control signal while the robotic arm is performing the picking task, and dynamically adjust the motion trajectory to avoid obstacles through an intelligent obstacle avoidance neural network model.

[0093] The specific functional implementation of each of the above modules can be found in the relevant contents of the method in Example 1 and will not be elaborated here.

[0094] Embodiment 3, this embodiment provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in Embodiment 1 are implemented.

[0095] Embodiment 4: This embodiment provides a computer device, including: Memory, for storing computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of any one of the methods described in Example 1.

[0096] Embodiment 5: This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of any method described in Embodiment 1 when executed by a processor.

[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0098] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A method for picking fruits and vegetables using a robotic arm based on embodied intelligence, characterized in that: include: Acquire fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes feature descriptions and picking requirements of the fruits and vegetables to be picked; According to the fruit and vegetable picking description information, a task instruction understanding model based on a large language model is used to analyze and generate a control signal for controlling the robotic arm; According to the control signal, the fruit and vegetable intelligent perception model of embodied intelligence is used to determine the category and grasping posture of the fruit and vegetable through the images collected by the depth camera, and the picking action is adjusted in combination with real-time feedback; According to the control signal, when the robotic arm performs the picking task, real-time environmental perception is performed through embodied intelligence technology, and the motion trajectory is dynamically adjusted through the intelligent obstacle avoidance neural network model to avoid obstacles.

2. The method for picking fruits and vegetables using a robotic arm based on embodied intelligence according to claim 1, characterized in that: The method of analyzing the fruit and vegetable picking description information by using a task instruction understanding model based on a large language model to generate a control signal for controlling the robotic arm includes: Receiving fruit and vegetable picking description information described in natural language input by a user; Convert the natural language description into multiple word vectors while maintaining contextual relationships; Using self-attention mechanism to identify important information in the description; Based on the pre-trained large language model, combined with the fine-tuning process, control signals are generated to control the robotic arm to perform picking tasks.

3. The method for picking fruits and vegetables using a robotic arm based on embodied intelligence according to claim 1, characterized in that: The fruit and vegetable intelligent perception model includes a Transformer layer, a convolutional neural network layer and a fully connected layer.

4. The method for picking fruits and vegetables using a robotic arm based on embodied intelligence according to claim 3, characterized in that: The fruit and vegetable intelligent perception model using embodied intelligence determines the category and grasping posture of fruits and vegetables through images captured by a depth camera, including: Performing feature extraction on the RGB image and the depth image through the Transformer layer of the fruit and vegetable intelligent perception model to obtain global visual features of the fruits and vegetables to be picked; Performing feature extraction on the RGB image and the depth image through the convolutional neural network layer of the fruit and vegetable intelligent perception model to obtain local visual features of the fruits and vegetables to be picked; According to the global visual features and the local visual features, the category of the picked fruits and vegetables and the grasping posture are determined through the fully connected layer of the fruit and vegetable intelligent perception model.

5. The method for picking fruits and vegetables using a robotic arm based on embodied intelligence according to claim 1, characterized in that: The real-time environment perception through embodied intelligence technology and the dynamic adjustment of the motion trajectory to avoid obstacles through the intelligent obstacle avoidance neural network model include: Controlling a plurality of depth cameras pre-set around the robotic arm to scan the activity space of the robotic arm at different angles, and obtaining voxel data of the physical environment at different angles around the robotic arm; Through multimodal sensor data fusion, the information collected by the visual sensor, depth sensor, and force sensor from different latitudes is collaboratively processed, and fused with the voxel data of the physical environment at different angles around the robotic arm, and input into the embodied intelligent perception model to extract features of the three-dimensional space. These multimodal features are fused through a weighted mechanism to finally obtain a comprehensive perception result and a three-dimensional space structure; Based on the comprehensive perception results and three-dimensional spatial structure, the motion trajectory is dynamically adjusted to avoid obstacles through the intelligent obstacle avoidance neural network model.

6. The method for picking fruits and vegetables using a robotic arm based on embodied intelligence according to claim 5, characterized in that: The architecture of the intelligent obstacle avoidance neural network model includes a three-dimensional convolutional layer, a Transformer layer, a force sensor information processing layer, a multimodal feature fusion layer, and a fully connected layer, which is used to dynamically perceive and reconstruct the three-dimensional physical environment around the robotic arm, and combine the force sensor feedback to adjust the motion path and grasping force of the robotic arm in real time.

7. A mechanical arm fruit and vegetable picking device based on embodied intelligence, characterized in that: include: A receiving module, used to obtain fruit and vegetable picking description information, wherein the fruit and vegetable picking description information includes a feature description and picking requirements of the fruits and vegetables to be picked; A processing module, configured to analyze the fruit and vegetable picking description information using a task instruction understanding model based on a large language model to generate a control signal for controlling the robotic arm; A perception module is used to determine the category and grasping posture of fruits and vegetables based on the control signal and the images captured by the depth camera using the embodied intelligent fruit and vegetable intelligent perception model, and adjust the picking action in combination with real-time feedback; The execution module is used to perform real-time environmental perception through embodied intelligence technology according to the control signal while the robotic arm is performing the picking task, and dynamically adjust the motion trajectory to avoid obstacles through an intelligent obstacle avoidance neural network model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer device, characterized in that: include: Memory, for storing computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images