Intelligent picking robot, method, terminal and medium

Abstract

The invention provides an intelligent picking robot and method, a terminal and a medium. A crop target monitoring and picking control model is constructed by adopting an RT-DETR target detection algorithm, an image captured by an image acquisition device is effectively processed, features are accurately extracted, crops in the image are identified, and a crop detection result is generated. According to the detection results, the model can intelligently command the mechanical arm to execute the corresponding picking action. According to the invention, high-speed and accurate picking of crops is realized, and wide adaptability to diversified fruits and vegetables is shown. And in a complex agricultural operation environment, a target picking object can be accurately identified and positioned, and a picking task can be efficiently completed, so that the overall fruit and vegetable picking efficiency and the commodity quality are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent picking control technology, and in particular to an intelligent picking robot, method, terminal and medium. Background Art

[0002] With the continuous increase in the global population and the increasing demand for high-quality agricultural products, the agricultural sector is accelerating its transformation towards modernization and intelligence. Traditional fruit and vegetable picking operations are highly dependent on manual labor. Although humans can effectively identify the maturity of fruits and implement picking with their keen judgment of vision and touch, this method exposes many limitations. On the one hand, high labor costs have become a major burden, especially in areas where labor resources are scarce. The strict timeliness and seasonality requirements of picking work have further pushed up costs. On the other hand, the efficiency bottleneck of manual picking can hardly meet the urgent needs of large-scale agricultural production. In addition, the risk of fruit damage during the picking process affects the commercial quality of agricultural products.

[0003] In response to the above challenges, the innovation of modern agricultural technology has promoted the rapid development of intelligent agricultural robot technology, which has shown broad application prospects in many aspects such as crop planting, management and harvesting. Especially in the field of fruit and vegetable picking, intelligent picking robots rely on integrated visual sensors, pressure sensors and other advanced technologies to achieve dynamic monitoring of the working environment and precise positioning of fruits. Through automated devices such as robotic arms, the automation level of picking operations has been effectively improved. However, current intelligent picking robots still face severe challenges in practical applications. In view of the wide variety of agricultural products, different maturity judgment standards, and significant differences in shape and size, existing robots are still insufficient in flexibility and adaptability. Especially in the rapid identification and precise positioning of agricultural products, the recognition accuracy and efficiency still need to be improved, and the risk of fruit damage still exists, which not only damages the quality and commercial value of the fruit, but also restricts the overall improvement of picking efficiency. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, the object of the present invention is to provide an intelligent picking robot, method, terminal and medium for solving technical problems such as slow picking speed, low precision, overlapping processing and poor effect on complex targets of the existing intelligent picking robots.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides an intelligent picking robot, comprising: a mechanical arm, fixed on a walking structure, for performing picking actions; an image acquisition device, for acquiring images of a picking environment in real time; a control device, connected to the mechanical arm and the image acquisition device, for performing target detection based on the images acquired by the image acquisition device based on a trained crop target monitoring and picking control model, and controlling the mechanical arm to perform corresponding picking operations; wherein the crop target monitoring and picking control model adopts an RT-DETR target detection algorithm, and the model comprises: a feature extraction module, for extracting features from the images acquired by the image acquisition device; a target detection module ... A detection module is connected to the feature extraction module and is used to detect crops appearing in the image based on the extracted features to obtain crop detection results; a decision module is connected to the target detection module and is used to control the robotic arm to perform corresponding picking operations according to the crop detection results; and wherein the feature extraction module includes: an efficient hybrid encoder; the efficient hybrid encoder includes: a feature preliminary extraction module, which is used to perform preliminary feature extraction on the input image; a feature interaction module based on a dynamic attention mechanism is connected to the feature extraction module and is used to dynamically adjust the parameters of the attention mechanism according to the preliminary extracted features, and obtain the features processed by the adjusted attention mechanism; a feature fusion module based on CNN is connected to the feature interaction module based on the dynamic attention mechanism and is used to fuse features of different scales processed by the attention mechanism through a series of fusion blocks.

[0006] In one embodiment of the present invention, the feature extraction module includes: an efficient hybrid encoder, including: a preliminary feature extraction module, used to perform preliminary feature extraction on the input image; a feature interaction module based on a dynamic attention mechanism, connected to the feature extraction module, used to dynamically adjust the parameters of the attention mechanism according to the preliminary extracted features, and obtain the features processed by the adjusted attention mechanism; a CNN-based feature fusion module, connected to the feature interaction module based on the dynamic attention mechanism, used to fuse features of different scales processed by the attention mechanism through a series of fusion blocks.

[0007] In one embodiment of the present invention, the feature interaction module based on the dynamic attention mechanism includes: a predictor using a lightweight network to analyze the input features and generate a set of parameters for dynamically adjusting the attention mechanism.

[0008] In one embodiment of the present invention, the target detection module includes: a query selection mechanism module and a transformer decoder; the query selection mechanism module is used to select features with small uncertainty based on the fused features and send them to the transformer decoder; the transformer decoder is connected to the query selection mechanism module and is used to iteratively optimize the features sent by the query selection mechanism module; the prediction module is connected to the transformer decoder and is used to generate detection categories and detection bounding boxes based on the features optimized by the transformer decoder to obtain crop detection results.

[0009] In one embodiment of the present invention, during the training of the crop target monitoring and harvesting control model, the intensity of data enhancement is adjusted according to the generalization ability of the model.

[0010] In one embodiment of the present invention, the control device further includes: a crop monitoring and analysis module, which is used to perform crop monitoring and data analysis based on the images of the picking environment collected in real time by the image acquisition device.

[0011] In one embodiment of the present invention, the intelligent harvesting robot further includes: an environmental perception module for adapting to different farmland environments and lighting conditions.

[0012] To achieve the above-mentioned purpose and other related purposes, the present invention provides an intelligent picking method, which is applied to an intelligent picking robot including: a mechanical arm and an image acquisition device, and the method includes: obtaining an image of a picking environment acquired by the image acquisition device in real time; based on a trained crop target monitoring and picking control model, performing target detection according to the image acquired by the image acquisition device, and controlling the mechanical arm to perform corresponding picking operations; wherein the crop target monitoring and picking control model adopts an RT-DETR target detection algorithm, and the model includes: a feature extraction module, which is used to extract features from the image acquired by the image acquisition device; a target detection module, connected to the feature extraction module, which is used to detect crops appearing in the image based on the extracted features and obtain crop detection results; a decision module, connected to the target detection module, which is used to control the mechanical arm to perform corresponding picking operations according to the crop detection results; and wherein the feature extraction module includes: an efficient hybrid encoder; the efficient hybrid encoder includes: a feature preliminary extraction module, which is used to perform preliminary feature extraction on the input image; a feature interaction module based on a dynamic attention mechanism, which is connected to the feature extraction module, which is used to dynamically adjust the parameters of the attention mechanism according to the preliminary extracted features and obtain the features processed by the adjusted attention mechanism; based on CNN The feature fusion module is connected to the feature interaction module based on the dynamic attention mechanism, and is used to fuse features of different scales processed by the attention mechanism through a series of fusion blocks.

[0013] To achieve the above-mentioned purpose and other related purposes, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store computer programs; the one or more processors are connected to the memories and are used to run the computer programs to execute the described method.

[0014] To achieve the above-mentioned object and other related objects, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by one or more processors to perform the method described.

[0015] As described above, the present invention is an intelligent picking robot, method, terminal and medium, which has the following beneficial effects: The present invention constructs a crop target monitoring and picking control model by adopting the RT-DETR target detection algorithm, effectively processes the image captured by the image acquisition device, accurately extracts features and identifies crops in the image, and generates crop detection results. Based on these detection results, the model can intelligently command the robotic arm to perform corresponding picking actions. The present invention not only realizes high-speed and accurate picking of crops, but also shows wide adaptability to diversified fruits and vegetables. In a complex agricultural operation environment, it can accurately identify and locate target picking objects, and efficiently complete picking tasks, thereby significantly improving the overall efficiency and commodity quality of fruit and vegetable picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Shown is a schematic diagram of the structure of an intelligent picking robot in one embodiment of the present invention.

[0017] Figure 2 Shown is a structural schematic diagram of a crop target monitoring and harvesting control model in one embodiment of the present invention.

[0018] Figure 3 Shown is a flow chart of an intelligent picking control method in one embodiment of the present invention.

[0019] Figure 4 Shown is a schematic structural diagram of an electronic terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments may also be used, and that mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the published patents. The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0022] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.

[0023] The terms first, second and third mentioned herein are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of the present invention.

[0024] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0025] The present invention provides an intelligent harvesting robot, which constructs a crop target monitoring and harvesting control model by adopting the RT-DETR target detection algorithm, effectively processes the image captured by the image acquisition device, accurately extracts features and identifies the crops in the image, and generates crop detection results. Based on these detection results, the model can intelligently command the robotic arm to perform corresponding harvesting actions. The present invention not only realizes high-speed and accurate harvesting of crops, but also shows wide adaptability to diversified fruits and vegetables. In a complex agricultural operation environment, it can accurately identify and locate target harvesting objects, and efficiently complete harvesting tasks, thereby significantly improving the overall efficiency and commodity quality of fruit and vegetable harvesting.

[0026] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0027] like Figure 1 A structural schematic diagram of an intelligent picking robot in an embodiment of the present invention is shown.

[0028] The intelligent picking robot comprises: A robotic arm 1 is fixed on the walking structure and is used to perform the picking action; the robotic arm 1 has multiple degrees of freedom, which gives it flexible movement ability and can freely adjust its position and posture in a complex picking environment. An actuator is installed at the end of the robotic arm 1, which can accurately perform the picking task according to the received picking control instructions. After the picking is completed, the actuator will properly place the harvested crops in the storage structure to ensure that the crops are not damaged during the picking process.

[0029] Image acquisition device 2 is responsible for collecting image information of the picking environment in real time. By continuously capturing images of the surrounding environment, it can provide detailed and accurate visual data for subsequent target detection and picking control.

[0030] The control device 3 is the core control unit of the intelligent harvesting robot and is connected to the robot arm 1 and the image acquisition device 2. It analyzes and processes the images collected by the image acquisition device based on the trained crop target monitoring and harvesting control model. Among them, the crop target monitoring and harvesting control model adopts the RT-DETR (Real-Time Detection Transformer) target detection algorithm. The RT-DETR algorithm is a target detection algorithm based on the Transformer structure. The algorithm has the characteristics of high efficiency and accuracy, and can significantly improve the performance of target detection. The model is further subdivided into the following three modules: The feature extraction module 311 is used to extract features from the images collected by the image acquisition device 2. By using specific algorithms and techniques, the module can extract key feature information related to crops from the images, providing basic data for subsequent target detection.

[0031] The target detection module 312 is connected to the feature extraction module 311, and detects crops appearing in the image based on the features extracted by the feature extraction module. This module can accurately identify the type and location information of the crops, thereby obtaining detailed crop detection results.

[0032] The decision module 313 is connected to the target detection module 312, and generates corresponding control instructions according to the crop detection result output by the target detection module 312, so as to control the robot arm 1 to perform corresponding picking operations.

[0033] The innovation of the present invention is to introduce the RT-DETR target detection algorithm into the field of crop target detection. Target detection algorithms have different performances in different application scenarios, and choosing the right algorithm is crucial for specific application fields. Crops present a variety of forms during their growth. In addition, crops usually grow densely in natural environments, which also increases the difficulty of target detection because crops may overlap or occlude each other. In crop target detection, traditional target detection algorithms may not be able to fully meet the needs, and the RT-DETR algorithm shows unique advantages.

[0034] The RT-DETR algorithm is more suitable for crop target detection. It may have more flexible feature extraction and representation capabilities, and can capture unique feature information of crops, such as texture, color, shape, etc. Compared with the RT-DETR algorithm applied to other fields, it shows better results in the field of crop target detection. This is because the algorithm considers unique features during the design and training process, or when optimized for crop target detection, it can better adapt to the complexity of crop morphology and the diversity of growth environment.

[0035] In one embodiment, if Figure 2 , the feature extraction module 311 includes: an efficient hybrid encoder, which is mainly composed of three parts: a feature preliminary extraction module, a feature interaction module based on a dynamic attention mechanism, and a feature fusion module based on CNN. These three parts are connected in sequence; The feature preliminary extraction module is used to perform preliminary feature extraction on the input image; specifically, feature extraction is a key step that converts the original image data (usually a matrix composed of pixels) into a more abstract and representative feature representation. These features can contain various information in the image, such as edges, textures, shapes, etc. By using various algorithms and techniques, this module can capture information in the image that is useful for subsequent tasks (such as object detection or classification). For example, in deep learning, commonly used feature extraction methods may involve some basic operations of convolutional neural networks (CNNs), such as convolution layers, pooling layers, etc.

[0036] The feature interaction module based on the dynamic attention mechanism is connected to the feature preliminary extraction module. Its uniqueness lies in the use of the dynamic attention mechanism. The attention mechanism is a technique in deep learning that allows the model to pay more "attention" to important parts when processing information. In this module, it dynamically adjusts the parameters of the attention mechanism according to the features obtained from the feature preliminary extraction module. The feature interaction module is used to dynamically adjust the parameters of the attention mechanism according to the extracted features and obtain the features processed by the adjusted attention mechanism; the principle is that different image regions have different complexities. For complex image regions, these regions may contain more information that needs attention, such as the boundary between the crop and the background, the details of the crop, etc., and the model needs more computing resources to accurately process this information; while for simple regions, such as large areas of solid color background, the model can reduce the allocation of computing resources. In this way, the module can improve the processing speed while ensuring the accuracy of the model, because it avoids over-calculation in simple areas. This part improves the feature interaction module of the traditional RT-DETR algorithm and adopts the dynamic attention mechanism. The traditional RT-DETR algorithm adopts a fixed attention mechanism or a simple weight allocation strategy in the feature interaction module. However, this static method cannot be flexibly adjusted according to the complexity of the input data. In contrast, the feature interaction module based on the dynamic attention mechanism can dynamically adjust the parameters of the attention mechanism according to the characteristics of the input data, thereby achieving efficient and accurate processing of image features.

[0037] The CNN-based feature fusion module is connected to the feature interaction module based on the dynamic attention mechanism, and is used to fuse the features of different scales processed by the attention mechanism through a series of fusion blocks; in deep learning, features of different scales contain different levels of information. For example, low-level features may contain more detailed information, such as textures and edges; while high-level features may contain more abstract information, such as the overall shape of an object. Through a series of fusion blocks (using convolution operations, upsampling and downsampling operations in CNN, etc.), these features of different scales can be fused to form a more comprehensive and expressive feature representation. Assuming that in a crop target monitoring scenario, the fusion of features of different scales allows the model to see both the local details of the crop (such as the texture of the leaves) and the overall outline of the crop, so as to more accurately identify the location, type and other information of the crop.

[0038] This feature extraction module 311 first extracts features from the original image through the collaboration of the three modules of the efficient hybrid encoder, then dynamically allocates attention based on the extracted features, and finally fuses the features processed by attention at different scales to provide more valuable and comprehensive feature information for subsequent tasks (such as target detection, classification or decision-making). This helps to improve the performance of the crop target monitoring and harvesting control model of the entire intelligent harvesting robot, making it smarter and more efficient when processing images.

[0039] In one embodiment, the feature interaction module based on the dynamic attention mechanism includes: a predictor using a lightweight network; the predictor inputs the initially extracted features. The main task of the predictor is to analyze the features of these inputs and generate a set of parameters, which will be used to dynamically adjust the attention mechanism. Dynamically adjusting the attention mechanism means that the behavior of the attention mechanism will be different according to the different features of the input data, thereby allocating computing resources and attention more flexibly.

[0040] The lightweight network used by the predictor can be a small convolution layer and a fully connected layer, etc. The small convolution layer can analyze the local features of the input features, which may help to discover the changes in local features and the relationship between features, and then generate parameters for adjusting the attention mechanism. For example, in an image, a small convolution layer can discover features such as texture changes and color distribution in different regions, which can be used as a basis for generating adjustment parameters. The fully connected layer can process the input features as a whole, extract the global relationship between features, and generate adjustment parameters related to the overall features. For example, the fully connected layer can determine which parts may be more important and which parts can be paid relatively less attention based on the overall distribution of the input features, thereby providing information for the adjustment of the attention mechanism.

[0041] In one embodiment, if Figure 2, the target detection module 312, comprises: a query selection mechanism module and a transformer decoder; A query selection mechanism module is used to select features with small uncertainty based on the fused features and send them to the transformer decoder; specifically, the uncertainty in this module is defined by the difference between the predicted positioning distribution (P) and the classification distribution (C). In the target detection task, the positioning distribution (P) usually involves the prediction of the location information of the target in the image, such as using a bounding box to represent the location range of the target. The classification distribution (C) involves the probability prediction of the target belonging to different categories.

[0042] This difference as a measure of uncertainty is based on the following considerations: when the difference between the location distribution and the classification distribution is large, it means that the model has a large uncertainty in predicting a certain area. For example, if for an image area, the model can neither determine whether the area contains an object (classification uncertainty) nor accurately determine the location of the object (location uncertainty), then the information in the area may be vague and unreliable; on the contrary, when the difference between the two is small, it means that the model has a clear judgment on the area, and its predictions are relatively stable and consistent for both the category and location of the object. During the training process, the model will select those features with the most consistent classification and location predictions as queries. This means that during the training process, the model will filter out those features that are more accurate in predicting both the category and the location based on its prediction results. In this way, the model can use the supervision information of the training data to find those features that are more likely to represent the target object and have accurate location, avoiding the use of vague or inaccurate feature information that may mislead subsequent processing. When the feature with the smallest uncertainty is used as the initial query, the decoder receives relatively reliable and high-quality information.

[0043] The transformer decoder is connected to the query selection mechanism module and is used to iteratively optimize the features sent by the query selection mechanism module; specifically, it receives the features sent by the query selection mechanism module and iteratively optimizes these features. In deep learning, iterative optimization is a common strategy. Through multiple iterations, the representation of features is gradually adjusted to make it more in line with the requirements of target detection. The transformer decoder can use its internal complex structure and parameters to refine the features to extract more useful information, improve the quality of features, and prepare for subsequent predictions. For example, in each iteration, it can adjust the weight of the features, enhance or suppress certain feature elements, so that the features are more focused on the target object and reduce the interference of noise and irrelevant information.

[0044] The prediction module is connected to the transformer decoder and generates detection categories and detection bounding boxes based on the features optimized by the transformer decoder. It comprehensively considers the optimized feature information and finally outputs the detection result, that is, determines the category and location information of the crop in the image.

[0045] In one embodiment, the training method of the crop target monitoring and picking control model includes: first, a set of image data of crops in a picking environment needs to be collected. The image source can be in various ways, such as actual data collected by an image acquisition device 2 installed on a picking robot, or it can be obtained from a public agricultural image data set, or it can be an image taken specifically for training the model. The collected sample images need to be labeled. For the target detection part, it is usually necessary to mark the location information of the crops in the image, generally represented by a bounding box, and also marked with a crop category label. Before the labeled sample data set is input into the model for training, some preprocessing operations are usually required. The preprocessed sample data set is input into the crop target monitoring and picking control model.

[0046] Preferably, the intensity of data enhancement is adjusted according to the generalization ability of the model during the training of the crop target monitoring and harvesting control model. Specifically, before the training starts, the initial data enhancement parameters are set according to experience or general standards. The model is trained using the training set, and the performance of the model is evaluated using the validation set after each training cycle. The intensity of data enhancement is dynamically adjusted according to the performance of the model on the validation set. The model is continued to be trained, and the evaluation and adjustment steps are repeated after each training cycle until the model performance reaches stability or a preset stop condition.

[0047] When the performance of the model on the validation set continues to improve, it means that the model is still learning new knowledge from the training data. At this time, the intensity of data enhancement can be appropriately increased to further enrich the diversity of training data and improve the generalization ability of the model. For example, the range of random cropping can be increased to enable the model to learn more local features; when the performance of the model on the validation set stagnates or decreases, it may be that the intensity of data enhancement is too large, making it difficult for the model to extract effective information from the training data. At this time, the intensity of data enhancement should be appropriately reduced. For example, an adjustment factor can be used to scale the parameters of data enhancement, reduce the range of random cropping, reduce the amplitude of color jitter, etc.

[0048] The crop target monitoring and picking control model of the present invention applies the dynamic attention mechanism to crop detection, and dynamically allocates computing resources according to the image content during feature extraction and processing, which is an innovative approach. It breaks through the problem of uneven resource allocation that may exist in traditional algorithms when processing crop images, avoids wasting computing resources on simple backgrounds, and the situation of insufficient resources in key crop areas, thereby improving the overall performance. The query selection mechanism with the least uncertainty is adopted, and the complexity and uncertainty in crop detection are taken into account. In a complex natural environment, there may be many uncertainties in the appearance and position of crops. By minimizing this uncertainty, the algorithm can improve the detection accuracy of crops, which is an improvement on the traditional target detection method. Combining multi-scale feature fusion and iterative optimization technology, the multi-scale feature information of crops is fully considered, and the feature quality is improved through iterative optimization, so that the model can better adapt to the different appearances and environmental changes of crops, and has better adaptability to naturally growing and diverse morphological targets such as crops, rather than simply using single-scale features for detection, thereby improving the detection ability of crops of different types and growth stages.

[0049] Traditional crop detection algorithms may have poor adaptability to complex crop morphology and changing environments. However, this model improves processing efficiency while ensuring detection accuracy through the above-mentioned innovative mechanisms of feature extraction and target detection, making it more suitable for large-scale, real-time crop monitoring and harvesting tasks. For example, traditional algorithms may have difficulty handling situations where crop fruits are partially obscured or under different lighting conditions, while the application of the RT-DETR algorithm in this model can more accurately identify these situations through multi-module collaboration, providing more reliable information for intelligent harvesting robots. Traditional algorithms may use relatively fixed feature processing and target detection methods, and perform poorly for the diversity of crops in different seasons, different growth stages, and different varieties. The innovative design of this model allows it to better handle these changes, and through dynamic adjustment and iterative optimization, it can better complete crop detection tasks as time and the environment change.

[0050] In one embodiment, the control device further includes: a crop monitoring and analysis module, which is used to perform crop monitoring and data analysis based on the image of the picking environment collected in real time by the image acquisition device; specifically, the crop monitoring and analysis module can analyze the color, shape, texture and other characteristics of the crop in the image, and the module can determine the growth status of the crop, such as whether it is mature, whether it is attacked by pests and diseases, etc., so as to monitor the growth status of the crop. This information is of great significance for taking picking or prevention measures in time. The crop monitoring and analysis module can also perform in-depth analysis on the collected image data, extract key indicators related to crop growth (such as plant height, leaf area index, fruit size, etc.), and generate detailed growth reports. These reports can provide support for precise management and decision-making of crops.

[0051] In one embodiment, the intelligent harvesting robot further includes: an environmental perception module for adapting to different farmland environments and lighting conditions. Specifically, the environmental perception module is a key system on the intelligent harvesting robot, which is responsible for collecting and analyzing real-time environmental information of the farmland where the robot is located. This information includes but is not limited to light intensity, soil moisture, temperature, wind speed, obstacle location, and crop distribution. By setting a light sensor in the environmental perception module, the light intensity in the farmland can be monitored in real time, and the robot's visual system and picking strategy can be adjusted according to the light changes. By setting radar, laser radar (LiDAR), ultrasonic sensor, etc. in the environmental perception module, the module can perceive obstacles (such as stones, branches, etc.) and crop distribution in the farmland in real time. By setting temperature sensors, humidity sensors, etc. in the environmental perception module, it is used to monitor the climatic conditions of the farmland. These data can help the robot predict the growth status of crops, adjust the picking time, and prevent mechanical failures or crop damage caused by extreme weather conditions.

[0052] Similar to the principle of the above embodiment, the present invention provides an intelligent picking control system.

[0053] The following provides specific embodiments in conjunction with the accompanying drawings: like Figure 3 A flow chart showing an intelligent picking control method in an embodiment of the present invention.

[0054] The intelligent picking robot includes: a mechanical arm and an image acquisition device, and the method includes: Step S1: an image acquisition device acquires images of the picking environment in real time; Step S2: Based on the trained crop target monitoring and picking control model, target detection is performed according to the image collected by the image acquisition device, and the robotic arm is controlled to perform corresponding picking operations; The crop target monitoring and harvesting control model adopts the RT-DETR target detection algorithm, and the model includes: A feature extraction module, used to extract features from the image captured by the image acquisition device; an object detection module, connected to the feature extraction module, for detecting crops appearing in the image based on the extracted features to obtain crop detection results; A decision module is connected to the target detection module and is used to control the robotic arm to perform corresponding picking operations according to the crop detection result.

[0055] Since the implementation principle of the intelligent picking control method has been described in the above embodiments, it will not be repeated here.

[0056] The intelligent picking control method provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 4 , is an optional hardware structure diagram of an electronic terminal 1000 provided in an embodiment of the present invention. The electronic terminal 1000 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The electronic terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 4 In the specification, various buses are labeled as bus systems.

[0057] The user interface 1009 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0058] It is understood that the memory 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0059] The memory 1002 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 1000. Examples of these data include: any executable program used to operate on the electronic terminal 1000, such as an operating system 10021 and an application 10022; the operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 10022 can include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The efficient management method for multi-threaded table entry indexes provided in the embodiment of the present invention can be included in the application 10022.

[0060] The method disclosed in the above embodiment of the present invention can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1001 or the instruction in the form of software. The above processor 1001 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The processor 1001 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0061] In an exemplary embodiment, the electronic terminal 1000 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0062] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0063] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0064] In summary, the intelligent picking robot, method, terminal and medium of the present invention, by adopting the RT-DETR target detection algorithm to construct a crop target monitoring and picking control model, effectively processes the image captured by the image acquisition device, accurately extracts features and identifies the crops in the image, and generates crop detection results. Based on these detection results, the model can intelligently command the robotic arm to perform corresponding picking actions. The present invention not only realizes high-speed and accurate picking of crops, but also shows wide adaptability to diversified fruits and vegetables. In a complex agricultural operation environment, it can accurately identify and locate the target picking object, and efficiently complete the picking task, thereby significantly improving the overall efficiency and commodity quality of fruit and vegetable picking. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0065] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An intelligent picking robot, characterized in that: include: A robotic arm, fixed to the walking structure, for performing a picking action; An image acquisition device, used for acquiring images of the picking environment in real time; A control device, connected to the mechanical arm and the image acquisition device, for performing target detection based on the images acquired by the image acquisition device based on the trained crop target monitoring and picking control model, and controlling the mechanical arm to perform corresponding picking operations; The crop target monitoring and harvesting control model adopts the RT-DETR target detection algorithm, and the model includes: A feature extraction module, used to extract features from the image captured by the image acquisition device; an object detection module, connected to the feature extraction module, for detecting crops appearing in the image based on the extracted features to obtain crop detection results; A decision module, connected to the target detection module, for controlling the robotic arm to perform corresponding picking operations according to the crop detection result; And wherein, the feature extraction module includes: an efficient hybrid encoder; the efficient hybrid encoder includes: a preliminary feature extraction module, used for preliminary feature extraction of the input image; a feature interaction module based on a dynamic attention mechanism, connected to the feature extraction module, used for dynamically adjusting the parameters of the attention mechanism according to the preliminary extracted features, and obtaining the features processed by the adjusted attention mechanism; a feature fusion module based on CNN, connected to the feature interaction module based on the dynamic attention mechanism, used for fusing features of different scales processed by the attention mechanism through a series of fusion blocks.

2. The intelligent picking robot according to claim 1, characterized in that: The feature interaction module based on the dynamic attention mechanism includes: a predictor using a lightweight network to analyze the input features and generate a set of parameters for dynamically adjusting the attention mechanism.

3. The intelligent picking robot according to claim 1, characterized in that: The target detection module includes: a query selection mechanism module and a transformer decoder; A query selection mechanism module, configured to select features with small uncertainty from the fused features and send them to the transformer decoder; a transformer decoder, connected to the query selection mechanism module, for iteratively optimizing the features sent by the query selection mechanism module; A prediction module is connected to the transformer decoder and is used to generate a detection category and a detection bounding box according to the features optimized by the transformer decoder to obtain a crop detection result.

4. The intelligent picking robot according to claim 1, characterized in that: During the training of the crop target monitoring and harvesting control model, the intensity of data enhancement is adjusted according to the generalization ability of the model.

5. The intelligent picking robot according to claim 1, characterized in that: The control device also includes: a crop monitoring and analysis module, which is used to perform crop monitoring and data analysis based on the images of the picking environment collected in real time by the image collection device.

6. The intelligent picking robot according to claim 1, characterized in that: The intelligent harvesting robot also includes: an environmental perception module for adapting to different farmland environments and lighting conditions.

7. An intelligent picking control method, characterized in that: The intelligent picking robot includes: a mechanical arm and an image acquisition device, and the method includes: The image acquisition device acquires images of the picking environment in real time; Based on the trained crop target monitoring and picking control model, target detection is performed according to the images collected by the image acquisition device, and the robotic arm is controlled to perform corresponding picking operations; The crop target monitoring and harvesting control model adopts the RT-DETR target detection algorithm, and the model includes: A feature extraction module, used to extract features from the image captured by the image acquisition device; an object detection module, connected to the feature extraction module, for detecting crops appearing in the image based on the extracted features to obtain crop detection results; A decision module, connected to the target detection module, for controlling the robotic arm to perform corresponding picking operations according to the crop detection result; And wherein, the feature extraction module includes: an efficient hybrid encoder; the efficient hybrid encoder includes: a preliminary feature extraction module, used for preliminary feature extraction of the input image; a feature interaction module based on a dynamic attention mechanism, connected to the feature extraction module, used for dynamically adjusting the parameters of the attention mechanism according to the preliminary extracted features, and obtaining the features processed by the adjusted attention mechanism; a feature fusion module based on CNN, connected to the feature interaction module based on the dynamic attention mechanism, used for fusing features of different scales processed by the attention mechanism through a series of fusion blocks.

8. An electronic terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors, connected to the memory, are configured to run the computer program to perform the method according to claim 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by one or more processors, the method according to claim 7 is performed.

Images