Apple identification and positioning method in automatic apple picking process

Abstract

The invention discloses an apple identification and positioning method in an automatic apple picking process, and relates to the technical field of computer vision, and the method comprises the steps: collecting apple pictures shot in a real apple orchard as an initial apple data set, and carrying out the amplification of the obtained initial apple data set; configuring a convolutional neural network and a YOLOv5s network, training based on the obtained initial apple data set, and constructing an apple recognition model; and processing the identified image through a median filtering method, constructing an apple positioning model, and processing apple positioning data. According to the method, the advanced convolutional neural network and the YOLOv5s network are configured, and the recognition result is post-processed in combination with the median filtering method, so that the apple positioning precision and stability are effectively improved. According to the method, high-precision apple recognition can be realized under complex illumination and shielding conditions, and the grabbing efficiency and the operation accuracy of the automatic picking robot can be remarkably improved through accurate three-dimensional positioning.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an apple recognition and positioning method in an automatic apple picking process. Background Art

[0002] In recent years, with the rapid development of agricultural automation and intelligent technology, automatic fruit picking systems have gradually become an important research direction in modern agriculture. In the field of apple picking, the traditional manual picking method has been difficult to meet the needs of large-scale production due to its high labor intensity, low efficiency and high cost. Therefore, automated picking technology based on computer vision and machine learning has emerged. Early apple recognition and positioning methods mainly relied on manually designed image feature extraction algorithms, such as color histograms, shape descriptors and texture analysis, which can achieve a certain degree of recognition effect under specific conditions. However, with the breakthrough of deep learning technology, especially the development of convolutional neural networks (CNN) and target detection algorithms such as YOLO (You Only Look Once), the accuracy and real-time performance of apple recognition and positioning have been significantly improved.

[0003] Although deep learning-based apple recognition and positioning methods have made significant progress in accuracy and efficiency, they still face many challenges in practical applications. First, the lighting conditions in real orchard environments are complex and changeable. Phenomena such as shadows, reflections, and light spots will significantly affect the quality of the image, and thus affect the performance of the recognition model. Secondly, the distribution of apples on fruit trees often has occlusion and overlap, and traditional single image recognition methods are difficult to effectively distinguish partially occluded or overlapping targets. In addition, when dealing with apples of different sizes and diverse shapes, existing recognition models often show a certain lack of generalization ability and are difficult to operate stably in a changing actual environment. More importantly, the existing methods have bottlenecks in positioning accuracy, making it difficult to achieve high-precision positioning of apples in three-dimensional space, which limits the grasping and operation efficiency of automatic picking robots. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an apple recognition and positioning method in the process of automatic apple picking to solve the problems of low apple recognition and positioning accuracy, low picking and sorting efficiency, and high time and labor costs in the process of automatic apple picking.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a method for identifying and locating apples during automatic apple picking, which includes collecting apple pictures taken in a real apple orchard as an initial apple data set, and amplifying the obtained initial apple data set; configuring a convolutional neural network and a YOLOv5s network, training based on the obtained initial apple data set, and constructing an apple recognition model; processing the recognized image by a median filtering method, constructing an apple positioning model, and processing the apple positioning data.

[0007] As a preferred solution of the apple recognition and positioning method in the automatic apple picking process of the present invention, wherein: collecting apple pictures taken in a real apple orchard as an initial apple data set includes the following steps: Analyze the features of apple images with shadows on the surface, apple images with uneven surface coloring, apple images with occlusions, and mixed and overlapped apple images; Apple images taken in real apple orchards are collected as the initial apple dataset.

[0008] As a preferred solution of the apple identification and positioning method in the automatic apple picking process of the present invention, the obtained initial apple data set is amplified, including the following steps: The initial apple data set is amplified by brightness transformation, contrast transformation, mirror transformation and fuzzy transformation; The Labelimg annotation tool was used to annotate the amplified apple dataset, delete unnecessary annotation information, and perform normalization. Convert the normalized annotation information into a txt format file.

[0009] As a preferred solution of the apple recognition and positioning method in the automatic apple picking process of the present invention, wherein: configuring a convolutional neural network and a YOLOv5s network includes the following steps: Use the PyTorch deep learning framework and its related libraries to build convolutional neural networks and YOLOv5s network structures, and use CSPDarknet53 as the backbone network of the YOLOv5s network; Introduce upsampling layers and connection layers in the Neck part of the YOLOv5s network to fuse feature maps at different levels; Multiple detection heads are set in the Head part of the YOLOv5s network to predict the location and category of the target.

[0010] As a preferred solution of the apple recognition and positioning method in the automatic apple picking process of the present invention, the following steps are included in the construction of an apple recognition model based on the obtained initial apple data set for training: The apple dataset converted into txt files is divided into training set, validation set and test set to train and optimize the model parameters; Train the YOLOv5s network model and use the training set to optimize the model parameters; Use the TensorBoard visualization tool to monitor and sample the recognition accuracy, recognition recall, and average detection precision in real time during training; Use data augmentation techniques to randomly rotate, scale, and crop images during training; An early stopping strategy is adopted to dynamically adjust the training rounds based on the mAP indicator of the validation set.

[0011] As a preferred solution of the apple recognition and positioning method in the automatic apple picking process of the present invention, the recognized image is processed by a median filtering method to construct an apple positioning model, including the following steps: The Orbbec Gemini depth camera based on binocular vision technology and structured light technology is used to obtain the depth information of the apple; Use OpenNI2 library to interface with Orbbec Gemini depth camera to obtain depth image data in real time; Combined with the OpenCV library, the acquired depth image data is preprocessed; The Orbbec Gemini depth camera based on binocular structured light technology projects coded light onto the surface of the apple through an infrared laser. The reflected light is collected by two infrared cameras on the left and right to obtain the depth information of the apple. The acquired depth image data is processed using a median filtering method; The processed depth image is fused with the object detection result of the apple recognition model to accurately determine the position of the apple in the three-dimensional space based on the two-dimensional coordinates and depth information of the apple in the image. Based on the coordinate system conversion formula, the two-dimensional image coordinates are converted into three-dimensional world coordinates to achieve the spatial positioning of the apple.

[0012] As a preferred solution of the method for identifying and locating apples in the process of automatic apple picking of the present invention, the preprocessing of the acquired depth image data includes graying, filtering and displaying the image.

[0013] As a preferred solution of the apple identification and positioning method in the automatic apple picking process of the present invention, the apple positioning data is processed, including the following steps: The Zhang Youzheng calibration method is used to calibrate the RGB camera to obtain the camera's intrinsic matrix and extrinsic parameters; According to the camera calibration parameters, the conversion between coordinate systems is performed to achieve the mapping of image pixel coordinates to world coordinates.

[0014] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: the computer program, when executed by the processor, implements any step of the method for identifying and locating apples in the automatic apple picking process as described in the first aspect of the present invention.

[0015] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: the computer program, when executed by a processor, implements any step of the method for identifying and locating apples in the automatic apple picking process as described in the first aspect of the present invention.

[0016] The beneficial effects of the present invention are as follows: by using motion blur to amplify the data set, the generalization ability of the model is enhanced while ensuring the accuracy of the camera in recognizing apples in motion, and a depth camera combining binocular vision technology and structured light technology is used to obtain the depth information of the image, and the depth information is integrated with the image recognition information to obtain the spatial position of the apple; thereby, accurate recognition and positioning of apples in the process of automatic apple picking is achieved, the efficiency of apple picking and sorting is improved, and time and labor costs are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them: Figure 1 This is a flow chart of the method for identifying and locating apples during the automatic apple picking process in Example 1.

[0018] Figure 2 This is a flow chart of camera calibration in Example 1. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the embodiments of the specification.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0022] Example 1 Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an apple identification and positioning method in the process of automatic apple picking, comprising the following steps: S1. Collect apple pictures taken in a real apple orchard as the initial apple dataset, including the following steps: In practical applications, there are many complex situations in apple images, mainly including the following aspects: Uneven lighting: Due to weather changes and different angles of sunlight, obvious light spots or shadows may appear on the surface of the apple. Uneven lighting causes obvious shadow areas on the surface of the apple.

[0023] Color change: Most ripe apples are red, but under different lighting conditions, the parts that are not fully exposed to the sun may be green, similar to the color of the background branches and leaves. This color change will affect the object detection algorithm's accurate recognition of apples.

[0024] Occlusion: Apples are often obscured by leaves and branches during their growth, causing partial or complete obscuration. Some apples are obscured by leaves or multiple apples overlap, increasing the difficulty of detection.

[0025] Apple images taken in real apple orchards are collected as the initial apple dataset.

[0026] Specifically, a high-resolution digital camera is used to ensure the clarity of image details, and an Orbbec Gemini binocular structured light depth camera is used to assist in obtaining depth information.

[0027] Select apple orchards in different seasons and weather conditions for image collection to capture the impact of light changes on the appearance of apples. Make sure the collection area covers fruit trees in different locations and arrangements in the orchard to avoid data bias.

[0028] Use handheld photography to photograph apples from multiple angles (such as the front, side, and top), covering different occlusions and arrangements. Ensure that each image contains 1-5 apples to avoid situations where they are too dense or too sparse. Record the shooting time, date, and ambient lighting conditions of each image to facilitate subsequent data analysis and model training.

[0029] The collected apple images are classified and stored according to the time sequence and shooting conditions to ensure the orderliness and traceability of the data. Blurred, unclear or repeated images are eliminated to ensure the quality of the data set.

[0030] It should be noted that the apple images collected above cover different lighting conditions (such as strong light, weak light), occlusion conditions (such as partial occlusion, complete occlusion) and apple sizes (small, medium, large, etc.), which lay a solid foundation for subsequent data expansion and model training.

[0031] S2, amplifying the obtained initial apple data set, including the following steps: The initial apple data set is amplified by brightness transformation, contrast transformation, mirror transformation and fuzzy transformation; Specifically, the brightness transformation is achieved by adjusting the Gamma value of the image. The specific operation is to multiply each pixel value of the image by the proportional coefficient of the brightness transformation, taking two levels of 0.5 and 1.5 respectively, to generate images with reduced brightness and enhanced brightness; By linearly stretching the contrast range of the image, the contrast transformation is achieved. The specific operation is to linearly adjust the pixel value of the image according to the proportional coefficient of the contrast transformation, taking two levels of 0.5 and 1.5 respectively, to generate images with reduced contrast and contrast enhanced contrast; Use image processing tools (such as OpenCV) to horizontally mirror each image and enlarge the image by 1 times to generate a mirror image.

[0032] Apply a motion blur filter to each image using OpenCV's cv2.blur or cv2.GaussianBlur function. The motion blur direction is randomly selected to simulate different motion directions and speeds.

[0033] Motion blur can be achieved through convolution operation, and its kernel function Defined as:

[0034] in, is the length of the blur kernel, usually adjusted according to the speed of the motion.

[0035] The Labelimg annotation tool was used to annotate the amplified apple dataset, delete unnecessary annotation information, and perform normalization.

[0036] Furthermore, download and install the Labelimg tool, configure the YOLO format annotation parameters, and ensure that the annotation results meet the training requirements of YOLOv5. Manually select the location of the apple in each image and generate a bounding box. Make sure that each bounding box tightly surrounds the target apple, especially in the case of partial occlusion, the entire apple area needs to be fully annotated. During the annotation process, record the category number of each bounding box (only apples are involved in this study, so the category number is unified as 0). Labelimg will save the annotation information as an XML file, including detailed information such as image size, target category, bounding box coordinates, etc.

[0037] Since YOLOv5 only needs the category number and bounding box coordinate information, the XML file generated by Labelimg needs to be optimized. Specifically, remove the information in the XML file that is not related to YOLOv5 training, such as image size, color channel, etc. Normalize the coordinates of the bounding box so that its value is between 0 and 1 to facilitate the input requirements of YOLOv5. The normalization formula is,

[0038] in, Indicates the position of the center of the target object on the horizontal axis of the image. Indicates the position of the target object's center on the vertical axis of the image, ( )and( ) are the coordinates of the upper left corner and lower right corner of the bounding box, and are the total width and height of the image, in pixels. is the width of the bounding box, is the height of the bounding box.

[0039] Use Python script to convert the optimized XML file into a txt file in YOLO format. Each line contains: category number, , , , .

[0040] S3, configure convolutional neural network and YOLOv5s network; Furthermore, in order to ensure the efficiency of model training and reasoning, high-performance computing equipment is selected as the recognition environment. The specific configuration is as follows: Processor (CPU): Intel(R) Core(TM) i7-13700KF, main frequency 3.40 GHz, with multi-core and multi-threading capabilities, able to efficiently handle parallel computing tasks.

[0041] Graphics card (GPU): NVIDIA GeForce RTX4090, 24GB video memory, supports Tensor cores, and accelerates the training and reasoning of deep learning models.

[0042] Memory (RAM): 32GB DDR5, ensuring large-scale data processing and storage requirements.

[0043] Storage: S70BLADE 2T M2 SSD, providing fast data reading and writing speeds, reducing data loading time.

[0044] The required software configuration is as follows: Operating system: Windows 10, providing a stable operating environment and wide software compatibility.

[0045] Development tools: PyCharm 2023.2.1, integrated development environment (IDE), supports efficient code writing and debugging.

[0046] Environment management: Anaconda3, which facilitates the management of Python environments and dependent libraries.

[0047] Programming language: Python 3.8, which is widely used in deep learning and data science.

[0048] Deep learning framework: PyTorch 2.0.1, a flexible and efficient deep learning framework that supports mixed training of dynamic and static graphs.

[0049] Related libraries: torch, torchvision: building and training deep learning models; numpy: supporting efficient numerical calculations; matplotlib, seaborn: data visualization tools; opencv-python: image processing and computer vision tasks; pandas: data processing and analysis; tqdm: displaying the training progress bar; tensorboard: visual monitoring of the model training process.

[0050] Use the PyTorch deep learning framework and its related libraries to build convolutional neural networks and YOLOv5s network structures, and use CSPDarknet53 as the backbone network of the YOLOv5s network to extract basic features of the image; In the Neck part of the YOLOv5s network, upsampling layers and connection layers are introduced to fuse feature maps of different levels, enhance the expression ability of features at different levels, and improve the detection performance; multiple detection heads are set in the Head part of the YOLOv5s network, which are responsible for predicting the position and category of the target and outputting the position coordinates and category probability of the prediction box.

[0051] It should be noted that the YOLOv5s model was selected as the target detection algorithm, which achieved a good balance between speed and accuracy and is suitable for apple detection tasks with high real-time requirements.

[0052] S4. Training based on the obtained initial apple data set to build an apple recognition model includes the following steps: The apple dataset converted into a txt file is divided into training set, validation set and test set in a ratio of 8:1:1; the training set is used to train the model parameters so that it can learn appropriate features and patterns from the data and improve the performance of the model on the training set; the validation set is used to adjust the model's hyperparameters and evaluate the model's performance to ensure that the model has good generalization ability and avoid overfitting; the test set is used to finally evaluate the performance of the model on unknown data and verify the generalization ability of the model.

[0053] Train the YOLOv5s network model and use the training set to optimize the model parameters; Specifically, to ensure the efficiency and accuracy of model training, the following training parameters are set: Number of training epochs: 1000, to ensure that the model fully learns the data features.

[0054] Batch Size: 32, balancing computing resources and training speed.

[0055] Input image size: 640×480, ensuring a balance between image details and computational efficiency.

[0056] Loss function: Use the default loss function of YOLOv5, including positioning loss ( ), confidence loss ( ) and classification loss ( ).

[0057] Load the dataset and ensure that the training set, validation set, and test set have been correctly divided and organized into the directory structure required by YOLOv5; the input image passes through the YOLOv5s network, calculates the loss between the predicted value and the true value, and uses the back propagation algorithm to update the model weights. The expression is,

[0058] in, is the positioning loss, which measures the difference between the predicted bounding box and the true bounding box. is the confidence loss, which measures the accuracy of target detection confidence. is the classification loss, which measures the accuracy of target category prediction.

[0059] Use the TensorBoard visualization tool to monitor and sample the indicators of recognition accuracy, recognition recall, and average detection accuracy in real time during the training process to ensure that the model gradually converges and improves performance during the training process. Furthermore, recognition accuracy measures the proportion of actual positive samples among the results predicted by the model as positive samples, recognition recall measures the proportion of the number of positive samples that the model can correctly detect to all actual positive samples, and average detection accuracy comprehensively considers the performance of the model at different recall rates and is an important indicator for measuring overall detection performance.

[0060] Use data augmentation techniques to randomly rotate, scale, and crop images during training; An early stopping strategy is adopted to dynamically adjust the training rounds based on the mAP (average detection accuracy) indicator of the validation set.

[0061] S5. Process the recognized image by median filtering to construct an apple positioning model, including the following steps: The Orbbec Gemini depth camera based on binocular vision technology and structured light technology is used to obtain depth information of the apple. The key parameters of the Orbbec Gemini depth camera include effective measurement range, resolution, and light adaptability. It should be noted that binocular structured light technology can provide accurate depth information under complex lighting conditions and shows higher reliability than traditional binocular vision or time-of-flight (TOF) technology.

[0062] Mount the Orbbec Gemini depth camera on the harvesting robot, making sure the camera lens is at the same level as the apple. Use a stable camera stand or mount to prevent camera displacement due to vibration or movement, ensuring accurate depth information collection. Adjust the distance between the camera and the apple to cover the working area where the apple is located. Usually keep it between 0.25m and 2.5m, depending on the effective measurement range of the camera. Make sure there are no obstacles in the camera's field of view to avoid errors in depth information collection and optimize the effect of apple detection and positioning.

[0063] Use OpenNI2 library to interface with Orbbec Gemini depth camera for communication and data acquisition, and obtain depth image data in real time; Combined with the OpenCV library, the acquired depth image data is grayed, filtered and displayed; The Orbbec Gemini depth camera based on binocular structured light technology projects coded light onto the surface of the apple through an infrared laser. The reflected light is collected by two infrared cameras on the left and right to obtain the depth information of the apple. The acquired depth image data is processed using the median filtering method to remove noise and missing depth values ​​and improve the accuracy of the depth information. Specifically, the size of a filter window is first determined, and the window is placed at a certain pixel position in the image. Then, the grayscale values ​​of all pixels in the window are extracted and sorted, and the sorted middle value is used as the new grayscale value of the current pixel position after filtering. Finally, the filter window is moved pixel by pixel on the image, and the above steps are repeated until the entire image is traversed.

[0064] The processed depth image is fused with the object detection result identified by the apple recognition model, and the position of the apple in the three-dimensional space is determined based on the two-dimensional coordinates and depth information of the apple in the image; Based on the coordinate system conversion formula, the two-dimensional image coordinates are converted into three-dimensional world coordinates to achieve the spatial positioning of the apple.

[0065] Furthermore, according to the apple detection results of the YOLOv5s model on the RGB image, the coordinates of the center point of each detection frame are extracted; the depth value of the center point of each detection frame is extracted from the filtered depth image, and the two-dimensional position of the apple in the image is converted into a three-dimensional position in the world coordinate system according to the obtained pixel coordinates and depth values, combined with the intrinsic and extrinsic parameters obtained by camera calibration. The expression is,

[0066] in, , is the center point coordinate in the image pixel coordinate system, , is the pixel coordinate of the center of the image, , is the focal length of the camera, , For pixels in Axis and Physical dimensions in the axial direction, , , is the translation vector of the camera coordinate system relative to the world coordinate system, is the depth value.

[0067] It should be noted that the world coordinate system represents the three-dimensional coordinate information of an object in the real world, in millimeters. For example, the actual spatial position of an apple is ; The camera coordinate system is the three-dimensional coordinate system of the camera itself, with the origin at the optical center of the camera, in millimeters, and the position of the object in the camera coordinate system ; The image pixel coordinate system is a two-dimensional pixel coordinate system on the imaging plane, with the unit being pixels. For example, the pixel position of a point in the image ; The image physical coordinate system is a two-dimensional physical coordinate system on the imaging plane, with units in millimeters. For example, the physical position of a point on the imaging plane .

[0068] The calculated world coordinates Stored in a database or directly transmitted to the control system of the picking robot for subsequent grasping and picking actions.

[0069] S6. Processing the apple location data includes the following steps: The RGB camera is calibrated using Zhang Youzheng's calibration method to obtain the camera's intrinsic parameter matrix and extrinsic parameters. Specifically, a checkerboard calibration template consisting of 6×9 black and white squares is used. The size of each small square on the calibration board is 25×25 mm to ensure that the checkerboard is clearly visible in the image. 26 calibration board images at different positions and angles are taken to ensure the diversity and coverage of the checkerboard in the image. Each image should contain a global checkerboard pattern without obvious distortion or occlusion. The Camera Calibrator toolbox is opened in MATLAB, the 26 collected calibration images are loaded, and the toolbox is used to automatically detect and extract the checkerboard corner points. Based on the extracted corner point positions, the camera's intrinsic parameter matrix and extrinsic parameters (rotation matrix and translation vector) are calculated. The camera's intrinsic parameter matrix It is expressed as,

[0070] in, , is the pixel coordinate of the center of the image.

[0071] Using the calibrated intrinsic and extrinsic parameters, the calibration image is converted between coordinate systems to achieve the mapping of image pixel coordinates to world coordinates.

[0072] This embodiment also provides a computer device, which is applicable to the method for identifying and locating apples during the automatic apple picking process, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to implement the method for identifying and locating apples during the automatic apple picking process as proposed in the above embodiment.

[0073] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0074] The present embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for identifying and locating apples in the process of automatic apple picking as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0075] In summary, the present invention collects apple pictures taken in a real apple orchard as the initial apple data set, and amplifies the obtained initial apple data set, ensuring that the apple recognition model has stronger generalization ability and robustness when facing various challenges in the real orchard environment, and significantly improves the recognition accuracy and stability of the automatic picking system in practical applications; through refined network structure design and efficient training process, the deep extraction and accurate recognition of apple features are realized, and the apple target can be detected efficiently and accurately, ensuring that the automatic picking robot can recognize the picking object in real time and reliably, thereby greatly improving the real-time performance and operation efficiency of the system; through median filtering, the noise and discontinuity in the depth map are effectively eliminated, and the accuracy of the depth data is improved. Subsequently, the processed depth information is integrated with the recognition result, and the two-dimensional image coordinates are mapped to the three-dimensional world coordinates through the coordinate system conversion formula, so as to realize the precise positioning of the apple in space. The beneficial effect of this step is that it ensures the high accuracy and stability of the positioning data, allowing the automatic picking robot to perform precise grasping and picking operations based on the three-dimensional spatial position, significantly improving the accuracy and efficiency of the picking process, while reducing grasping failures or misgrasping caused by positioning errors.

[0076] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for identifying and locating apples during automatic apple picking, characterized in that: include, Apple pictures taken in real apple orchards are collected as the initial apple dataset, and the obtained initial apple dataset is augmented; Configure the convolutional neural network and the YOLOv5s network, train them based on the initial apple dataset, and build an apple recognition model. The recognized image is processed by the median filtering method, the apple positioning model is constructed, and the apple positioning data is processed.

2. The method for identifying and locating apples during automatic apple picking according to claim 1, characterized in that: Collecting apple pictures taken in real apple orchards as the initial apple dataset includes the following steps: Analyze the features of apple images with shadows on the surface, apple images with uneven surface coloring, apple images with occlusions, and mixed and overlapped apple images; Apple images taken in real apple orchards are collected as the initial apple dataset.

3. The method for identifying and locating apples during automatic apple picking as claimed in claim 2, characterized in that: The initial apple dataset is augmented, including the following steps: The initial apple data set is amplified by brightness transformation, contrast transformation, mirror transformation and fuzzy transformation; The Labelimg annotation tool was used to annotate the amplified apple dataset, delete unnecessary annotation information, and perform normalization. Convert the normalized annotation information into a txt format file.

4. The method for identifying and locating apples during automatic apple picking as claimed in claim 3, characterized in that: Configuring the convolutional neural network and the YOLOv5s network includes the following steps: Use the PyTorch deep learning framework and its related libraries to build convolutional neural networks and YOLOv5s network structures, and use CSPDarknet53 as the backbone network of the YOLOv5s network; Introduce upsampling layers and connection layers in the Neck part of the YOLOv5s network to fuse feature maps at different levels; Multiple detection heads are set in the Head part of the YOLOv5s network to predict the location and category of the target.

5. The method for identifying and locating apples during automatic apple picking as claimed in claim 4, characterized in that: Based on the initial apple data set obtained, training and building an apple recognition model include the following steps: The apple dataset converted into txt files is divided into training set, validation set and test set to train and optimize the model parameters; Train the YOLOv5s network model and use the training set to optimize the model parameters; Use the TensorBoard visualization tool to monitor and sample the recognition accuracy, recognition recall, and average detection precision in real time during training; Use data augmentation techniques to randomly rotate, scale, and crop images during training; An early stopping strategy is adopted to dynamically adjust the training rounds based on the mAP indicator of the validation set.

6. The method for identifying and locating apples during automatic apple picking as claimed in claim 5, characterized in that: The recognized image is processed by the median filtering method to construct an apple positioning model, which includes the following steps: The Orbbec Gemini depth camera based on binocular vision technology and structured light technology is used to obtain the depth information of the apple; Use OpenNI2 library to interface with Orbbec Gemini depth camera to obtain depth image data in real time; Combined with the OpenCV library, the acquired depth image data is preprocessed; The Orbbec Gemini depth camera based on binocular structured light technology projects coded light onto the surface of the apple through an infrared laser. The reflected light is collected by two infrared cameras on the left and right to obtain the depth information of the apple. The acquired depth image data is processed using a median filtering method; The processed depth image is fused with the object detection result identified by the apple recognition model, and the position of the apple in the three-dimensional space is determined based on the two-dimensional coordinates and depth information of the apple in the image; Based on the coordinate system conversion formula, the two-dimensional image coordinates are converted into three-dimensional world coordinates to achieve the spatial positioning of the apple.

7. The method for identifying and locating apples during automatic apple picking as claimed in claim 6, characterized in that: The preprocessing of the acquired depth image data includes graying, filtering and displaying the image.

8. The method for identifying and locating apples during automatic apple picking as claimed in claim 7, characterized in that: Processing Apple location data includes the following steps: The Zhang Youzheng calibration method is used to calibrate the RGB camera to obtain the camera's intrinsic matrix and extrinsic parameters; According to the camera calibration parameters, the conversion between coordinate systems is performed to achieve the mapping of image pixel coordinates to world coordinates.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for identifying and locating apples in the automatic apple picking process described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying and locating apples in the automatic apple picking process described in any one of claims 1 to 7 are implemented.

Images