A leafy vegetable harvesting system and method based on deep learning

Abstract

The present invention provides a leaf vegetable harvesting system and method based on deep learning, and relates to the field of image processing technology. The method comprises: collecting images to be harvested; inputting the images to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps: calling a first weed image set, identifying the first weed on the image to be harvested, marking the position of the first weed, and obtaining a first marked image to be harvested; generating a clearing instruction according to the position of the first weed; calling a second weed image set, identifying the second weed on the first marked image to be harvested, marking the position of the second weed, and obtaining a second marked image to be harvested; calling a leaf vegetable image set, identifying the leaf vegetables on the second marked image to be harvested, extracting the leaf vegetable position adjacent to the second weed position, and obtaining a leaf vegetable extraction position; determining a blank cutting area according to the second weed position and the leaf vegetable extraction position, and generating a harvesting instruction. Weeds are identified in stages, image processing is reduced step by step, and recognition speed is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a leafy vegetable harvesting system and method based on deep learning. Background Art

[0002] Weeds have long been a problem for agricultural producers in the field growing leafy vegetables. Due to the complex field environment and the diverse variety of weeds, their growth cycles and speeds vary. This not only complicates field management but also poses a serious threat to the growth and yield of leafy vegetables. Some weeds closely resemble leafy vegetables, making manual differentiation extremely difficult and further exacerbating the weed management challenge.

[0003] To improve the efficiency and accuracy of field management, modern agriculture has begun to incorporate intelligent technologies, including the use of mechanical equipment for harvesting leafy vegetables. However, during mechanized harvesting, effectively identifying and removing weeds has become a pressing technical challenge. To address this, researchers have developed a weed recognition algorithm that automatically distinguishes weeds from leafy vegetables by processing and analyzing image data collected from the field.

[0004] However, weed recognition algorithms face numerous challenges in practical application. Due to the complex and ever-changing field environment, the amount of image data collected and processed is enormous. Furthermore, excessive algorithm complexity leads to a sharp increase in computing resource consumption, which in turn reduces recognition speed and fails to meet the real-time requirements of mechanized harvesting. This not only reduces harvesting efficiency but can also lead to confusion between leafy vegetables and weeds due to delayed recognition, compromising harvest quality.

[0005] On the other hand, reducing the complexity of the algorithm in order to increase the recognition speed may lead to a decrease in recognition accuracy. Because weeds and leafy vegetables are similar in appearance, the algorithm needs to have high precision and robustness when distinguishing them. Once the algorithm is too simple, it will be difficult to accurately capture and distinguish the subtle differences between the two, resulting in an increase in the misrecognition rate. Summary of the Invention

[0006] The purpose of the present invention is to provide a leafy vegetable harvesting system and method based on deep learning. The technical problem to be solved is how to improve the recognition speed while ensuring the recognition accuracy to meet the real-time requirements of mechanized harvesting.

[0007] The present invention is achieved through the following technical solutions:

[0008] A first aspect provides a leafy vegetable harvesting method based on deep learning, comprising the following steps:

[0009] S100: After receiving the harvesting task, collect images of the harvesting area to be harvested;

[0010] S200: Input the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps:

[0011] S210: calling a first weed image set, identifying a first weed on the to-be-harvested image using the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;

[0012] generating a removal instruction according to a first weed position in the first to-be-harvested marked image;

[0013] S220: Calling a second weed image set, identifying a second weed on the first marked image to be harvested using the second weed image set, marking a position of the second weed on the first marked image to be harvested, and obtaining a second marked image to be harvested;

[0014] S230: Retrieving a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image using the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;

[0015] S240, determining a blank area for cutting based on the second weed position and the leafy vegetable extraction position in the second marked image to be harvested;

[0016] Generate harvesting instructions based on the above-mentioned cutting blank area.

[0017] A phased recognition strategy is employed, first identifying prominent weeds using a first weed image set, followed by further identification using a second weed image set. The first phase quickly filters out most non-leafy areas, reducing the amount of image data required for subsequent processing. Phased recognition allows for the use of more targeted models and datasets at each phase, thereby improving recognition accuracy. Each image set is optimized for a specific target (e.g., weeds or leafy vegetables), making the model more efficient at identifying these targets. After identifying weeds and leafy vegetables in each phase, removal and harvesting instructions are generated, respectively. By specifying the locations of weeds to be removed and areas of leafy vegetables to be harvested, precise control of harvesting locations is achieved, reducing the amount of weeds mixed into the leafy vegetables after harvesting. Phased recognition and the use of targeted image sets reduce unnecessary computation and indirectly reduce algorithm complexity, thereby improving overall recognition speed and meeting the real-time requirements of mechanized harvesting. Furthermore, in actual use, the execution time difference between the first and second phase instructions is sufficient to delay weed identification, reducing the risk of misidentification and providing strong support for agricultural automation and intelligentization.

[0018] Furthermore, before training the above weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include:

[0019] S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables;

[0020] S002. Converting the leafy vegetable image and the weed image into feature vectors to obtain a leafy vegetable feature vector and a weed feature vector;

[0021] S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector;

[0022] S004, determining the similarity between the leafy vegetable image and the weed image using the leafy vegetable unit vector and the weed unit vector;

[0023] S005. Call the similarity threshold and compare the above similarity with the similarity threshold;

[0024] When the similarity is less than the similarity threshold, the weed image is classified as the first weed image;

[0025] When the similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.

[0026] By collecting images of weeds mixed with leafy vegetables, we can ensure that the dataset contains a variety of possible weed types, growth stages, and relative positional relationships with leafy vegetables, thereby improving the generalization ability of the model. A dataset containing rich weed samples helps the model learn to distinguish the characteristics of weeds from leafy vegetables, reducing the false recognition rate in practical applications. Converting images into feature vectors allows the model to extract useful information from the original image data. Normalizing the feature vectors to ensure that all feature vectors are on the same scale. Normalized feature vectors are more efficient in calculation because they are more numerically uniform, reducing errors and unnecessary calculations in the calculation process. By calculating the similarity between the leaf unit vector and the weed unit vector, the degree of similarity between them is quantified, and the weed images are classified using a similarity threshold.

[0027] Furthermore, the similarity between the leafy vegetable image and the weed image is determined using the following formula:

[0028] ;

[0029] in, Represents similarity, with a value range of [-1, 1], where 1 represents complete similarity, -1 represents complete oppositeness, and 0 represents no correlation; Indicates the leaf menu bit vector of dimensions; Indicates the Weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.

[0030] The above formula calculates the cosine similarity between two vectors by calculating the sum of the dot products of the leaf unit vector and the weed unit vector in each dimension and dividing it by the product of their respective moduli. Cosine similarity measures the degree of similarity between two vectors in direction, not their size or length. Cosine similarity only focuses on the direction of the vector, not its size. Because unit vectors are used, implicit normalization is performed, which helps to reduce computational errors caused by different scales of feature vectors. The calculation of cosine similarity is relatively simple, involving only basic vector operations, and therefore has high computational efficiency when processing large-scale datasets.

[0031] Furthermore, using the following formula, the specific steps of training the above weed recognition model include:

[0032] S006. Acquire a plurality of historical harvest images and historical marked images, wherein the historical marked images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; and associate the historical harvest images with the historical marked images.

[0033] S007, dividing the plurality of historical harvest images obtained above into a training set and a test set, and using the obtained historical marked images as a verification set;

[0034] S008. Calling a neural network model and configuring parameters of the neural network model;

[0035] Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position;

[0036] S009. Retrieving a historically marked image from the validation set, associating the historically marked image with the historical harvest image input into the neural network model, and updating the parameters of the neural network model by matching the historically marked image with the output image to obtain a trained weed recognition model.

[0037] Obtain historical data: First, collect multiple historical harvest images and corresponding historical marked images. These historical marked images are images after the first weed position and the second weed position are manually marked on the original historical harvest images; ensure that each historical harvest image is correctly associated with its corresponding historical marked image to facilitate comparison of the output results and the actual results; divide the acquired historical harvest images into training sets and test sets. The training set is used to train the model, and the test set is used to evaluate the performance of the model after training is completed; use the historical marked images as the validation set. The validation set is used to adjust the parameters of the model during training to ensure that the model performs well on unseen data; select a suitable neural network model as the basis for weed recognition. This model can be a convolutional neural network (CNN), a recurrent neural network (RNN) or other models suitable for image recognition; configure the parameters of the neural network model, including the number of layers, the number of neurons, the learning rate, the optimizer, etc.; The historical harvest images in the training set are input into the neural network model, and the model will output an image with the first and second weed positions marked (the model will output two images in succession, the first image only marks the first weed position, and the second image marks both the first and second weed positions, so the output image at this time is the second image). The difference between the output image and the true labeled image (that is, the historical labeled image in the validation set) is calculated. The loss function is usually used to measure this difference. Then, the parameters of the model are updated through the backpropagation algorithm to reduce the value of the loss function. The above process is repeated until the performance of the model on the validation set reaches the preset standard or the number of training times reaches the upper limit. After training is completed, the performance of the model is evaluated using the test set, which helps to understand the generalization ability of the model on unseen data. According to the performance on the test set, the parameters of the model can be further adjusted. After sufficient training and evaluation, a trained weed recognition model is obtained.

[0038] A second aspect provides a leafy vegetable harvesting system, which employs the above-mentioned harvesting method;

[0039] The harvesting system includes:

[0040] A mobile carrier, wherein the mobile carrier is provided with a first positioning module, and the first positioning module is used to locate the position of the mobile carrier to obtain a first positioning coordinate;

[0041] A collection module is provided in front of the mobile carrier and is used to collect images of the harvested area after receiving a harvesting task; a second positioning module is provided on the collection module and is used to locate the position of the collection module and obtain a second positioning coordinate;

[0042] A processing module is connected to the acquisition module, the first positioning module and the second positioning module, and is used to perform the following steps:

[0043] The image to be harvested is input into the trained weed recognition model, which then performs the following steps:

[0044] calling a first weed image set, identifying a first weed on the to-be-harvested image using the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;

[0045] generating and sending a removal instruction according to the position of the first weed in the first marked image to be harvested;

[0046] calling a second weed image set, identifying a second weed on the first to-be-harvested marked image using the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;

[0047] Recalling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image using the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed position from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;

[0048] Determining a blank area for cutting based on the second weed position and the leafy vegetable extraction position in the second marked image to be harvested;

[0049] Generate harvesting instructions based on the above-mentioned cutting blank area;

[0050] When the first positioning coordinates move to the second positioning coordinates, a harvesting instruction is sent;

[0051] The execution module is connected to the processing module. After receiving the clearing instruction, the execution module clears the weeds at the first weed position, and after receiving the harvesting instruction, the execution module harvests the leafy vegetables in the harvesting area.

[0052] The above-mentioned mobile carrier serves as the mobile platform of the entire harvesting system and is responsible for moving in the farmland to perform the harvesting task; the acquisition module is set in front of the mobile carrier to ensure that the image to be harvested in front can be acquired in real time during the movement; the execution module is connected to the processing module and is responsible for executing the instructions from the processing module. The positioning modules are respectively set on the mobile carrier and the acquisition module to determine the current positions of the mobile carrier and the acquisition module; after generating a clearing instruction, the processing module immediately sends the clearing instruction to the execution module to clear the marked first weeds; after generating a harvesting instruction, it is temporarily not sent to the execution module. When the first positioning coordinates coincide with the second positioning coordinates or specific conditions are met, the harvesting instruction is sent to the execution module.

[0053] Furthermore, the execution module includes a plucking mechanism, a cutting mechanism and a transport mechanism. The plucking mechanism is arranged in front of the mobile carrier, and the collection module is connected to the plucking mechanism.

[0054] One end of the transport mechanism is connected to the front of the mobile carrier, and the other end extends to a side away from the mobile carrier and is inclined downward along the extension direction;

[0055] The cutting mechanism is arranged at one end of the transport mechanism away from the mobile carrier, and is used for cutting leafy vegetables.

[0056] The above-mentioned pulling mechanism is arranged directly in front of the mobile carrier, as the front end part of the execution module, directly facing the farmland area to be harvested; after receiving the clearing instruction, the pulling mechanism will start and remove the marked first weed (which can be achieved by a robotic arm or similar device, which can accurately locate and pull out the weeds while avoiding damage to the surrounding leafy vegetables); the collection module is connected to the pulling mechanism. While pulling out the weeds, the collection module can continue to collect images of the weeds to be harvested in front, providing data support for subsequent leafy vegetable identification; the main function of the transportation mechanism is to transport the cut leafy vegetables from the harvesting area to the designated collection point; the cutting mechanism is arranged at the end of the transportation mechanism away from the mobile carrier, and the cutting mechanism and the transportation mechanism work closely together to ensure that the cut leafy vegetables can fall directly on the transportation mechanism and then be transported to the collection point along the inclined track.

[0057] Furthermore, the cutting mechanism includes a support shaft, and both ends of the support shaft are provided with connecting pieces, which are connected to the transport mechanism;

[0058] The support shaft is provided with a plurality of blades, which reciprocate along the axial direction of the support shaft, and adjacent blades move in opposite directions;

[0059] The above processing module is used to configure the blade code. Each blade has a unique code, and the corresponding code is determined by cutting the blank area.

[0060] When the cutting mechanism is used to execute a harvesting instruction, the blade corresponding to the code stops moving toward the cutting blank area.

[0061] The above-mentioned support shaft is the core component of the cutting mechanism. The support shaft not only provides stable support for the blades, but also ensures that they can reciprocate along the axial direction; the connecting parts are located at both ends of the support shaft, which are used to connect the support shaft and the transport mechanism, ensuring that the cutting mechanism can be firmly installed at the end of the transport mechanism and move with the movement of the mobile carrier; multiple blades are evenly distributed along the support shaft, and each blade can independently reciprocate along the axial direction of the support shaft, so that the cutting mechanism can handle multiple cutting tasks at the same time, significantly improving the cutting efficiency; adjacent blades move in opposite directions. When two adjacent blades move relative to each other, the space between them will gradually shrink, thereby achieving the cutting of leafy vegetables; when two adjacent blades move back to back, the space between them will gradually increase, forming a blank area; the processing module configures a unique code for each blade. These codes are not only used to identify and manage blades, but also can determine the blade that needs to stop moving according to the information of the cutting blank area when executing the harvesting instruction; when the processing module issues a harvesting instruction, it will calculate the code of the blade that needs to stop moving based on the information of the cutting blank area. Then, the control system sends a stop signal to the corresponding blade to ensure that weeds in the blank area are not cut during the cutting process.

[0062] Furthermore, the removal mechanism includes a fixing member and a removal member, one end of the fixing member is connected to the movable carrier, and the other end of the fixing member is provided with a connecting rod, and the connecting rod is provided with a slide rail;

[0063] One end of the plucking member is connected to the slide rail, and the plucking member moves along the slide rail; the other end of the plucking member is provided with a clamping portion, and the clamping portion is used to clamp the first weeds.

[0064] The above-mentioned fixing part serves as a supporting component of the plucking mechanism. One end of the fixing part is connected to the mobile carrier, ensuring the stability and reliability of the plucking mechanism; a connecting rod is provided at the other end of the fixing part, which not only connects the fixing part and the plucking part, but also provides a moving path for the plucking part; a slide rail is provided on the connecting rod, which provides a guide for the movement of the plucking part. The plucking part moves on the slide rail to achieve the plucking of weeds; one end of the plucking part is connected to the slide rail and can move smoothly along the slide rail, ensuring that the plucking part can accurately locate the position of the weeds; a clamping part is provided on the plucking part, which usually adopts a mechanical clamping claw or a similar device to clamp the weeds to ensure that they will not fall off during the plucking process.

[0065] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0066] A phased recognition strategy is employed, first identifying prominent weeds using a first weed image set, followed by further identification using a second weed image set. The first phase quickly filters out most non-leafy areas, reducing the amount of image data required for subsequent processing. Phased recognition allows for the use of more targeted models and datasets at each phase, thereby improving recognition accuracy. Each image set is optimized for a specific target (e.g., weeds or leafy vegetables), making the model more efficient at identifying these targets. After identifying weeds and leafy vegetables in each phase, removal and harvesting instructions are generated, respectively. By specifying the locations of weeds to be removed and areas of leafy vegetables to be harvested, precise control of harvesting locations is achieved, reducing the amount of weeds mixed into the leafy vegetables after harvesting. Phased recognition and the use of targeted image sets reduce unnecessary computation and indirectly reduce algorithm complexity, thereby improving overall recognition speed and meeting the real-time requirements of mechanized harvesting. Furthermore, in actual use, the execution time difference between the first and second phase instructions is sufficient to delay weed identification, reducing the risk of misidentification and providing strong support for agricultural automation and intelligentization. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0068] Figure 1 This is the main flow chart of the harvesting method;

[0069] Figure 2 It is the connection diagram of the harvesting system;

[0070] Figure 3 It is the overall plan diagram of the harvesting system;

[0071] Figure 4 It is a structural diagram of the cutting mechanism.

[0072] Markings and corresponding parts names in the accompanying drawings:

[0073] 1. Mobile carrier; 2. Collection module; 3. Execution module; 31. Transport mechanism; 32. Cutting mechanism; 321. Support shaft; 322. Connecting piece; 323. Blade; 33. Extraction mechanism; 331. Fixing piece; 332. Connecting rod; 333. Extraction piece; 334. Clamping part; 4. Processing module; 5. First positioning module; 6. Second positioning module. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0075] First embodiment:

[0076] Combine Figure 1 , a leafy vegetable harvesting method based on deep learning, comprising the following steps:

[0077] S100: After receiving the harvesting task, collect images of the harvesting area to be harvested;

[0078] S200: Input the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps:

[0079] S210: calling a first weed image set, identifying a first weed on the to-be-harvested image using the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;

[0080] generating a removal instruction according to a first weed position in the first to-be-harvested marked image;

[0081] S220: Calling a second weed image set, identifying a second weed on the first marked image to be harvested using the second weed image set, marking a position of the second weed on the first marked image to be harvested, and obtaining a second marked image to be harvested;

[0082] S230: Retrieving a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image using the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;

[0083] S240, determining a blank area for cutting based on the second weed position and the leafy vegetable extraction position in the second marked image to be harvested;

[0084] Generate harvesting instructions based on the above-mentioned cutting blank area.

[0085] A phased recognition strategy is employed, first identifying prominent weeds using a first weed image set, followed by further identification using a second weed image set. The first phase quickly filters out most non-leafy areas, reducing the amount of image data required for subsequent processing. Phased recognition allows for the use of more targeted models and datasets at each phase, thereby improving recognition accuracy. Each image set is optimized for a specific target (e.g., weeds or leafy vegetables), making the model more efficient at identifying these targets. After identifying weeds and leafy vegetables in each phase, removal and harvesting instructions are generated, respectively. By specifying the locations of weeds to be removed and areas of leafy vegetables to be harvested, precise control of harvesting locations is achieved, reducing the amount of weeds mixed into the leafy vegetables after harvesting. Phased recognition and the use of targeted image sets reduce unnecessary computation and indirectly reduce algorithm complexity, thereby improving overall recognition speed and meeting the real-time requirements of mechanized harvesting. Furthermore, in actual use, the execution time difference between the first and second phase instructions is sufficient to delay weed identification, reducing the risk of misidentification and providing strong support for agricultural automation and intelligentization.

[0086] Second embodiment:

[0087] Based on the first embodiment, before training the above weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include:

[0088] S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables;

[0089] S002. Converting the leafy vegetable image and the weed image into feature vectors to obtain a leafy vegetable feature vector and a weed feature vector;

[0090] S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector;

[0091] S004, determining the similarity between the leafy vegetable image and the weed image using the leafy vegetable unit vector and the weed unit vector;

[0092] S005. Call the similarity threshold and compare the above similarity with the similarity threshold;

[0093] When the similarity is less than the similarity threshold, the weed image is classified as the first weed image;

[0094] When the similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.

[0095] By collecting images of weeds mixed with leafy vegetables, we can ensure that the dataset contains various possible weed types, growth stages, and relative positional relationships with leafy vegetables, thereby improving the generalization ability of the model; a dataset containing rich weed samples helps the model learn to distinguish the characteristics of weeds and leafy vegetables, reducing the false recognition rate in practical applications; converting images into feature vectors allows the model to extract useful information from the original image data; normalizing the feature vectors to ensure that all feature vectors are on the same scale. The normalized feature vectors are more efficient in calculation because they are more numerically uniform, reducing errors and unnecessary calculations in the calculation process; by calculating the similarity between the leaf unit vector and the weed unit vector, the degree of similarity between them is quantified, and the weed images are classified using a similarity threshold.

[0096] Dividing weed images into a first and second weed image set helps the model better learn the characteristics of different weed types during training. While this step does not directly reduce algorithm complexity, optimizing the dataset makes the model more efficient during training, indirectly improving recognition speed and meeting the real-time requirements of mechanized harvesting. Models trained using carefully constructed datasets have stronger generalization capabilities in practical applications and can better adapt to various complex field environments.

[0097] In a specific embodiment, the similarity between the leafy vegetable image and the weed image is determined using the following formula:

[0098] ;

[0099] in, Represents similarity, with a value range of [-1, 1], where 1 represents complete similarity, -1 represents complete oppositeness, and 0 represents no correlation; Indicates the leaf menu bit vector of dimensions; Indicates the Weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.

[0100] The above formula calculates the cosine similarity between two vectors by calculating the sum of the dot products of the leaf unit vector and the weed unit vector in each dimension and dividing it by the product of their respective moduli. Cosine similarity measures the degree of similarity between two vectors in direction, not their size or length. Cosine similarity only focuses on the direction of the vector, not its size. Because unit vectors are used, implicit normalization is performed, which helps to reduce computational errors caused by different scales of feature vectors. The calculation of cosine similarity is relatively simple, involving only basic vector operations, and therefore has high computational efficiency when processing large-scale datasets.

[0101] An application scenario for reference, assuming that the leafy vegetable image and the weed image are converted into feature vectors and normalized to obtain the leafy vegetable feature vector and the weed unit vector Contains 4 dimensions, such as 、 ,in, 、 、 and Represents the leaf menu bit vectors of the 1st, 2nd, 3rd and 4th dimensions respectively , 、 、 and Represents the weed unit vectors of the 1st, 2nd, 3rd and 4th dimensions respectively ; First calculate the leaf menu bit vector and the weed unit vector The dot product of , and then calculate the leaf menu bit vector and the weed unit vector Length of mold and , and finally calculate the cosine similarity When the cosine similarity is less than the similarity threshold, the weed image is classified as the first weed image; when the above similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.

[0102] Third embodiment:

[0103] Based on any of the above embodiments, the specific steps of training the above weed recognition model using the following formula include:

[0104] S006. Acquire a plurality of historical harvest images and historical marked images, wherein the historical marked images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; and associate the historical harvest images with the historical marked images.

[0105] S007, dividing the plurality of historical harvest images obtained above into a training set and a test set, and using the obtained historical marked images as a verification set;

[0106] S008. Calling a neural network model and configuring parameters of the neural network model;

[0107] Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position;

[0108] S009. Retrieving a historically marked image from the validation set, associating the historically marked image with the historical harvest image input into the neural network model, and updating the parameters of the neural network model by matching the historically marked image with the output image to obtain a trained weed recognition model.

[0109] Obtain historical data: First, collect multiple historical harvest images and corresponding historical marked images. These historical marked images are images after the first weed position and the second weed position are manually marked on the original historical harvest images; ensure that each historical harvest image is correctly associated with its corresponding historical marked image to facilitate comparison of the output results and the actual results; divide the acquired historical harvest images into training sets and test sets. The training set is used to train the model, and the test set is used to evaluate the performance of the model after training is completed; use the historical marked images as the validation set. The validation set is used to adjust the parameters of the model during training to ensure that the model performs well on unseen data; select a suitable neural network model as the basis for weed recognition. This model can be a convolutional neural network (CNN), a recurrent neural network (RNN) or other models suitable for image recognition; configure the parameters of the neural network model, including the number of layers, the number of neurons, the learning rate, the optimizer, etc.; The historical harvest images in the training set are input into the neural network model, and the model will output an image with the first and second weed positions marked (the model will output two images in succession, the first image only marks the first weed position, and the second image marks both the first and second weed positions, so the output image at this time is the second image). The difference between the output image and the true labeled image (that is, the historical labeled image in the validation set) is calculated. The loss function is usually used to measure this difference. Then, the parameters of the model are updated through the backpropagation algorithm to reduce the value of the loss function. The above process is repeated until the performance of the model on the validation set reaches the preset standard or the number of training times reaches the upper limit. After training is completed, the performance of the model is evaluated using the test set, which helps to understand the generalization ability of the model on unseen data. According to the performance on the test set, the parameters of the model can be further adjusted. After sufficient training and evaluation, a trained weed recognition model is obtained.

[0110] Fourth embodiment:

[0111] Combine Figure 2 , a leafy vegetable harvesting system, the harvesting system adopting the above-mentioned harvesting method;

[0112] The harvesting system includes:

[0113] A mobile carrier 1 is provided with a first positioning module 5, and the first positioning module 5 is used to locate the position of the mobile carrier 1 to obtain a first positioning coordinate;

[0114] The acquisition module 2 is arranged in front of the mobile carrier 1. The acquisition module 2 is used to collect images of the harvested area after receiving the harvesting task. The acquisition module 2 is provided with a second positioning module 6. The second positioning module 6 is used to locate the position of the acquisition module 2 and obtain a second positioning coordinate.

[0115] The processing module 4 is connected to the acquisition module 2, the first positioning module 5 and the second positioning module 6. The processing module 4 is used to perform the following steps:

[0116] The image to be harvested is input into the trained weed recognition model, which then performs the following steps:

[0117] calling a first weed image set, identifying a first weed on the to-be-harvested image using the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;

[0118] generating and sending a removal instruction according to the position of the first weed in the first marked image to be harvested;

[0119] calling a second weed image set, identifying a second weed on the first to-be-harvested marked image using the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;

[0120] Recalling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image using the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed position from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;

[0121] Determining a blank area for cutting based on the second weed position and the leafy vegetable extraction position in the second marked image to be harvested;

[0122] Generate harvesting instructions based on the above-mentioned cutting blank area;

[0123] When the first positioning coordinates move to the second positioning coordinates, a harvesting instruction is sent;

[0124] The execution module 3 is connected to the processing module 4. After receiving the clearing instruction, the execution module 3 clears the weeds at the first weed position, and after receiving the harvesting instruction, the execution module 3 harvests the leafy vegetables in the harvesting area.

[0125] The above-mentioned mobile carrier 1 serves as the mobile platform of the entire harvesting system and is responsible for moving in the farmland to perform the harvesting task; the acquisition module 2 is set directly in front of the mobile carrier 1 to ensure that the image to be harvested in front can be collected in real time during the movement; the execution module 3 is connected to the processing module 4 and is responsible for executing the instructions from the processing module 4. Positioning modules are respectively set on the mobile carrier 1 and the acquisition module 2 to determine the current positions of the mobile carrier 1 and the acquisition module 2; after generating a clearing instruction, the processing module 4 immediately sends the clearing instruction to the execution module 3 to clear the marked first weeds; after generating a harvesting instruction, it is temporarily not sent to the execution module 3. When the first positioning coordinates coincide with the second positioning coordinates or meet specific conditions, the harvesting instruction is sent to the execution module 3.

[0126] Fifth embodiment:

[0127] Based on the fourth embodiment, combined with Figure 3 The execution module 3 includes a plucking mechanism 33, a cutting mechanism 32 and a transport mechanism 31. The plucking mechanism 33 is arranged in front of the mobile carrier 1, and the collection module 2 is connected to the plucking mechanism 33;

[0128] One end of the transport mechanism 31 is connected to the front of the mobile carrier 1, and the other end extends to a side away from the mobile carrier 1 and is inclined downward along the extension direction;

[0129] The cutting mechanism 32 is arranged at one end of the transport mechanism 31 away from the mobile carrier 1, and is used to cut leafy vegetables; the cutting mechanism 32 runs synchronously with the transport mechanism 31; the pulling mechanism 33 extends from the end of the cutting mechanism 32 and is located above the cutting mechanism 32.

[0130] The above-mentioned pulling out mechanism 33 is arranged in front of the mobile carrier 1, as the front end part of the execution module 3, directly facing the farmland area to be harvested; after receiving the clearing instruction, the pulling out mechanism 33 will start and remove the marked first weeds (which can be achieved by a robotic arm or similar device, which can accurately locate and pull out the weeds while avoiding damage to the surrounding leafy vegetables); the collection module 2 is connected to the pulling out mechanism 33. While pulling out the weeds, the collection module 2 can continue to collect the images to be harvested in front, providing data support for subsequent leafy vegetable identification; the main function of the transportation mechanism 31 is to transport the cut leafy vegetables from the harvesting area to the designated collection point; the cutting mechanism 32 is arranged at the end of the transportation mechanism 31 away from the mobile carrier 1, and the cutting mechanism 32 cooperates closely with the transportation mechanism 31 to ensure that the cut leafy vegetables can fall directly on the transportation mechanism 31 and then be transported to the collection point along the inclined track.

[0131] The workflow of the harvesting system:

[0132] (1) The mobile carrier 1 moves in the farmland, and the weeding mechanism 33 removes the weeds according to the clearing instruction, while the acquisition module 2 continues to collect the image to be harvested in front;

[0133] (2) The processing module 4 receives the image to be harvested and calls the trained weed recognition model to identify weeds and leafy vegetables; based on the recognition results, it generates a removal instruction and a harvesting instruction and sends them to the execution module 3;

[0134] (3) The plucking mechanism 33 removes weeds according to the clearing instruction, while the cutting mechanism 32 cuts leafy vegetables in the designated blank area according to the harvesting instruction;

[0135] (4) The cut leafy vegetables fall onto the transport mechanism 31 and are then transported to a collection point along an inclined track.

[0136] Sixth embodiment:

[0137] Based on the fifth embodiment, combined Figure 4 The cutting mechanism 32 includes a support shaft 321 , and connecting members 322 are provided at both ends of the support shaft 321 , and the connecting members 322 are connected to the transport mechanism 31 ;

[0138] The support shaft 321 is provided with a plurality of blades 323. Each blade 323 can be connected to a driver, which can be nested inside the support shaft 321. The driver is connected to the processing module 4 and is used to drive the blades 323 to move according to the harvesting instructions. The driver controls the blades 323 to reciprocate along the axial direction of the support shaft 321, and adjacent blades 323 move in opposite directions.

[0139] The processing module 4 is used to configure the blade 323 code (i.e., the driver code). Each blade 323 has a unique code, and the corresponding code is determined by cutting the blank area.

[0140] When the cutting mechanism 32 is used to execute a harvesting instruction, the blade 323 corresponding to the code stops moving toward the cutting blank area.

[0141] The support shaft 321 is the core component of the cutting mechanism 32. The support shaft 321 not only provides stable support for the blades 323, but also ensures that they can reciprocate along the axial direction. The connecting members 322 are located at both ends of the support shaft 321 and are used to connect the support shaft 321 with the transport mechanism 31, ensuring that the cutting mechanism 32 can be firmly installed at the end of the transport mechanism 31 and move with the movement of the mobile carrier 1. The multiple blades 323 are evenly distributed along the support shaft 321, and each blade 323 can independently reciprocate along the axial direction of the support shaft 321, so that the cutting mechanism 32 can handle multiple cutting tasks at the same time, significantly improving the cutting efficiency. Adjacent blades 323 move in opposite directions. When two adjacent blades 323 move relative to each other, the space between them will gradually shrink, thereby achieving the cutting of leafy vegetables. When two adjacent blades 323 move in opposite directions, the space between them will gradually increase, forming a blank area. The processing module 4 configures a unique code for each blade 323. These codes are not only used to identify and manage the blades 323, but also can determine the blade 323 that needs to stop moving based on the information of the blank area when executing the harvesting instruction. When the processing module 4 issues a harvesting instruction, it will calculate the code of the blade 323 that needs to stop moving based on the information of the blank area. Then, the control system sends a stop signal to the corresponding blade 323 to ensure that weeds in the blank area are not cut during the cutting process.

[0142] The working process of the processing module 4 and the cutting mechanism 32 is as follows:

[0143] The processing module 4 determines the code of the blade 323 that needs to stop moving according to the blank area for cutting and the code of the configured blade 323;

[0144] The cutting mechanism 32 performs the cutting operation according to the harvesting instruction, and the blade 323 corresponding to the code stops moving when reaching the cutting blank area, thereby reducing the cutting of the second weeds.

[0145] Through the design of the cutting mechanism 32, the leafy vegetable harvesting system can not only efficiently remove weeds and collect leafy vegetables, but also control the movement of the blade 323 according to the actual cutting blank area information to reduce the cutting of the second weeds.

[0146] Seventh embodiment:

[0147] Based on the fifth embodiment, combined Figure 3 The removal mechanism 33 includes a fixing member 331 and a removal member 333. One end of the fixing member 331 is connected to the mobile carrier 1. The other end of the fixing member 331 is provided with a connecting rod 332. The connecting rod 332 is provided with a slide rail (not shown). Figure 3 (drawn in);

[0148] One end of the plucking member 333 is connected to the slide rail, and the plucking member 333 moves along the slide rail; the other end of the plucking member 333 is provided with a clamping portion 334, and the clamping portion 334 is used to clamp the first weed.

[0149] The fixing member 331 is used as a supporting member of the plucking mechanism 33. One end of the fixing member 331 is connected to the mobile carrier 1, ensuring the stability and reliability of the plucking mechanism 33. The other end of the fixing member 331 is provided with a connecting rod 332. The connecting rod 332 not only connects the fixing member 331 and the plucking member 333, but also provides a moving path for the plucking member 333. The slide rail is provided on the connecting rod 332 to provide a guide for the movement of the plucking member 333. The plucking member 333 moves on the slide rail to achieve the plucking of weeds. One end of the member 333 is connected to the slide rail and can move smoothly along the slide rail, ensuring that the plucking member 333 can accurately locate the position of the weeds; the plucking member 333 is provided with a clamping portion 334, which usually adopts a mechanical clamping claw or a similar device, and is driven by a driver to achieve clamping. There is no specific limitation on the clamping portion 334, the driver and the slide rail. It is only used to realize that after the clamping portion 334 clamps the weeds, the plucking member 333 moves along the slide rail, pulls the weeds to pull them out, and throws the weeds out of the harvesting area.

[0150] The working process of the extraction mechanism 33:

[0151] The processing module 4 can determine the actual coordinates of the second weeds based on the position of the second weeds marked on the image to be harvested and the second positioning coordinates, and determine the moving path of the plucking member 333 through the actual coordinates of the second weeds and the current coordinates of the plucking member 333; the plucking member 333 moves to the designated position along the connecting rod 332 under the guidance of the slide rail; the clamping portion 334 closes and clamps the weeds after reaching the designated position; after clamping the weeds, the plucking member 333 moves along the slide rail to pull the weeds out of the soil; after plucking the weeds, the clamping portion 334 opens and releases the weeds, and then the plucking member 333 returns to the initial position to prepare for the next plucking operation; it is worth noting that the moving speed of the mobile carrier 1 during harvesting is usually slow, and there is no need to consider the problem that the plucking member 333 cannot keep up with the moving speed when the moving speed is too fast.

[0152] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A leafy vegetable harvesting method based on deep learning, characterized in that: The following steps are involved: S100, after receiving the harvesting task, collecting the image to be harvested in the harvesting area; S200: Input the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps: S210, calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image; generating a removal instruction according to a first weed position in the first to-be-harvested marked image; S220, calling a second weed image set, identifying a second weed on the first to-be-harvested marked image through the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image; S230, calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions; S240, determining a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image; Generate a harvesting instruction according to the cutting blank area; Before training the weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include: S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables; S002, converting the leafy vegetable image and the weed image into feature vectors to obtain a leafy vegetable feature vector and a weed feature vector; S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector; S004, determining the similarity between the leafy vegetable image and the weed image through the leafy vegetable unit vector and the weed unit vector; S005, calling a similarity threshold, and comparing the similarity with the similarity threshold; When the similarity is less than the similarity threshold, classifying the weed image as a first weed image; When the similarity is equal to or greater than the similarity threshold, classifying the weed image as a second weed image; Using the following formula, the specific steps of training the weed recognition model include: S006, acquiring a plurality of historical harvest images and historical marking images, wherein the historical marking images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; and associating the historical harvest images with the historical marking images; S007, dividing the plurality of historical acquisition images obtained into a training set and a test set, and using the obtained historical marked images as a verification set; S008. Calling a neural network model and configuring parameters of the neural network model; Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position; S009, calling a historically marked image from the verification set, associating the historically marked image with the historical harvest image input into the neural network model, and updating the parameters of the neural network model by matching the historically marked image with the output image to obtain a trained weed recognition model.

2. The leafy vegetable harvesting method according to claim 1, characterized in that: Use the following formula to determine the similarity between leafy vegetable images and weed images: ； in, Indicates similarity; Indicates The leaf menu bit vector of dimensions; Indicates The weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.

3. A leafy vegetable harvesting system, characterized in that: The harvesting system adopts the harvesting method according to claim 1 or 2; The harvesting system includes: A mobile carrier (1), wherein a first positioning module (5) is provided on the mobile carrier (1), and the first positioning module (5) is used to locate the position of the mobile carrier (1) to obtain a first positioning coordinate; A collection module (2), the collection module (2) being arranged in front of the mobile carrier (1), the collection module (2) being used to collect the image to be collected in the collection area after receiving the collection task; a second positioning module (6) being arranged on the collection module (2), the second positioning module (6) being used to locate the position of the collection module (2) to obtain a second positioning coordinate; A processing module (4), the processing module (4) is connected to the acquisition module (2), the first positioning module (5) and the second positioning module (6), and the processing module (4) is used to perform the following steps: The image to be harvested is input into a trained weed recognition model, and the weed recognition model performs the following steps: calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image; generating and sending a removal instruction according to a first weed position in the first to-be-harvested marked image; calling a second weed image set, identifying second weeds on the first to-be-harvested marked image through the second weed image set, marking a position of the second weeds on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image; calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions; Determine a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image; Generate a harvesting instruction according to the cutting blank area; When the first positioning coordinate moves to the second positioning coordinate, sending a harvesting instruction; An execution module (3) is connected to the processing module (4), and after receiving a clearing instruction, the execution module (3) clears the weeds at the first weed position, and after receiving a harvesting instruction, the execution module (3) harvests the leafy vegetables in the harvesting area.

4. The leafy vegetable harvesting system according to claim 3, characterized in that: The execution module (3) comprises a plucking mechanism (33), a cutting mechanism (32) and a transport mechanism (31); the plucking mechanism (33) is arranged directly in front of the mobile carrier (1); and the collection module (2) is connected to the plucking mechanism (33); One end of the transport mechanism (31) is connected to the front of the mobile carrier (1), and the other end extends to a side away from the mobile carrier (1) and is inclined downward along the extension direction; The cutting mechanism (32) is arranged at one end of the transport mechanism (31) away from the mobile carrier (1), and the cutting mechanism (32) is used for cutting leafy vegetables.

5. The leafy vegetable harvesting system according to claim 4, characterized in that: The cutting mechanism (32) comprises a supporting shaft (321), and connecting pieces (322) are provided at both ends of the supporting shaft (321), and the connecting pieces (322) are connected to the transport mechanism (31); A plurality of blades (323) are arranged on the support shaft (321), and the blades (323) reciprocate along the axial direction of the support shaft (321), and adjacent blades (323) move in opposite directions; The processing module (4) is used to configure the blade (323) code, each blade (323) has a unique code, and the corresponding code is determined by cutting the blank area; When the cutting mechanism (32) is used to execute a harvesting instruction, the blade (323) corresponding to the code stops moving toward the cutting blank area.

6. The leafy vegetable harvesting system according to claim 4, characterized in that: The pulling mechanism (33) comprises a fixing member (331) and a pulling member (333), one end of the fixing member (331) is connected to the movable carrier (1), and the other end of the fixing member (331) is provided with a connecting rod (332), and a slide rail is provided on the connecting rod (332); One end of the plucking member (333) is connected to the slide rail, and the plucking member (333) moves along the slide rail; the other end of the plucking member (333) is provided with a clamping portion (334), and the clamping portion (334) is used to clamp the first weed.

Images