Leaf vegetable harvesting system and method based on deep learning
Through the use of phased identification strategies and targeted image sets, combined with mobile carriers and execution modules, the problem of difficult to take into account the speed and accuracy of weed recognition in the existing technology is solved, and efficient and accurate leafy vegetables harvest is achieved, satisfying the satisfaction of Real-time requirements for mechanized harvesting.
Patent Information
- Application Number
- CN202411570893.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-11-06
Smart Images

Figure CN119091365A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a leafy vegetable harvesting system and method based on deep learning. Background Art
[0002] In the process of growing leafy vegetables in the field, weeds have always been a problem that has troubled agricultural producers. Due to the complex field environment, there are many types of weeds, and their growth cycles and speeds are also different, which not only increases the difficulty of field management, but also poses a serious threat to the growth and yield of leafy vegetables. Some weeds are very similar in appearance to leafy vegetables, making it extremely difficult to distinguish them manually, further exacerbating the problem of weed management.
[0003] In order to improve the efficiency and accuracy of field management, modern agriculture has begun to introduce intelligent technologies, including the use of mechanical equipment to harvest leafy vegetables. However, in the process of mechanized harvesting, how to effectively identify and remove weeds has become a technical problem that needs to be solved urgently. To this end, researchers have developed a weed recognition algorithm that can automatically distinguish between weeds and leafy vegetables by processing and analyzing image data collected in the field.
[0004] However, weed recognition algorithms also face many challenges in practical applications. Due to the complex and changeable field environment, the amount of image data collected and processed is very large. On the one hand, when the complexity of the algorithm is too high, it will lead to a sharp increase in the consumption of computing resources, which will reduce the recognition speed and fail to meet the real-time requirements of mechanized harvesting. It will not only reduce the harvesting efficiency, but also may cause confusion between leafy vegetables and weeds due to recognition delays, affecting the harvesting quality.
[0005] On the other hand, if the complexity of the algorithm is reduced in order to increase the recognition speed, it may lead to a decrease in recognition accuracy. Because of the similarity in appearance between weeds and leafy vegetables, the algorithm needs to have high precision and robustness when distinguishing between them. Once the algorithm is too simple, it will be difficult to accurately capture and distinguish the subtle differences between the two, resulting in an increase in the misrecognition rate. Summary of the invention
[0006] The purpose of the present invention is to provide a leafy vegetable harvesting system and method based on deep learning. The technical problem to be solved is how to improve the recognition speed while ensuring the recognition accuracy to meet the real-time requirements of mechanized harvesting.
[0007] The present invention is achieved through the following technical solutions:
[0008] A first aspect provides a leafy vegetable harvesting method based on deep learning, comprising the following steps:
[0009] S100, after receiving the harvesting task, collecting the image to be harvested in the harvesting area;
[0010] S200, inputting the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps:
[0011] S210, calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;
[0012] generating a removal instruction according to the first weed position in the first to-be-harvested marked image;
[0013] S220, calling a second weed image set, identifying a second weed on the first to-be-harvested marked image through the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;
[0014] S230, calling a leafy vegetable image set, identifying leafy vegetables on the second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;
[0015] S240, determining a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image;
[0016] According to the above-mentioned cutting blank area, a harvesting instruction is generated.
[0017] A phased recognition strategy is adopted. Obvious weeds are first identified through the first weed image set, and then further identified through the second weed image set. The first phase of recognition can quickly filter out most of the non-leafy areas, reducing the amount of image data to be processed later; phased recognition allows the use of more targeted models and data sets at each stage, thereby improving the accuracy of recognition; each image set is optimized for specific targets (such as weeds or leafy vegetables), making the model more efficient in identifying these targets; after weeds and leafy vegetables are identified at each stage, clearing instructions and harvesting instructions are generated respectively. By clearly specifying the locations of weeds to be cleared and the areas of leafy vegetables that do not need to be harvested, precise control of the harvesting position is achieved, reducing the amount of weeds mixed into leafy vegetables after harvesting; the use of phased recognition and targeted image sets reduces unnecessary calculations, indirectly reduces the complexity of the algorithm, thereby improving the overall recognition speed and meeting the real-time requirements of mechanized harvesting. In addition, in actual use, there is a time difference between the execution of the first and second phase instructions, which is sufficient to support the delay in weed recognition, reduce the risk of misidentification, and provide strong support for agricultural automation and intelligence.
[0018] Furthermore, before training the above weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include:
[0019] S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables;
[0020] S002, converting the leafy vegetable images and the weed images into feature vectors to obtain leafy vegetable feature vectors and weed feature vectors;
[0021] S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector;
[0022] S004, determining the similarity between the leafy vegetable image and the weed image through the leafy vegetable unit vector and the weed unit vector;
[0023] S005, calling the similarity threshold, and comparing the above similarity with the similarity threshold;
[0024] When the similarity is less than the similarity threshold, the weed image is classified as the first weed image;
[0025] When the similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.
[0026] By collecting images of weeds mixed with leafy vegetables, it is possible to ensure that the dataset contains various possible weed types, growth stages, and relative positional relationships with leafy vegetables, thereby improving the generalization ability of the model; a dataset containing rich weed samples helps the model learn to distinguish the features of weeds and leafy vegetables, reducing the misrecognition rate in practical applications; converting images into feature vectors allows the model to extract useful information from the original image data; normalizing the feature vectors to ensure that all feature vectors are on the same scale. Normalized feature vectors are more efficient in calculation because they are more uniform in numerical value, reducing errors and unnecessary calculations in the calculation process; by calculating the similarity between the leaf unit vector and the weed unit vector, the degree of similarity between them is quantified, and the weed images are classified by the similarity threshold;
[0027] Furthermore, the similarity between the leafy vegetable image and the weed image is determined using the following formula:
[0028] ;
[0029] in, Represents similarity, with a value range of [-1, 1], 1 represents complete similarity, -1 represents complete opposite, and 0 represents no correlation; Indicates The leaf menu bit vector of dimensions; Indicates The weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.
[0030] The above formula calculates the sum of the dot products of the leaf unit vector and the weed unit vector in each dimension and divides it by the product of their respective moduli to obtain the cosine similarity between two vectors. Cosine similarity measures the degree of similarity between two vectors in direction, not their size or length. Cosine similarity only focuses on the direction of the vector, not its size. Since unit vectors are used, normalization is implicit, which helps to reduce calculation errors caused by different scales of feature vectors. The calculation of cosine similarity is relatively simple and only involves basic vector operations, so it has high computational efficiency when processing large-scale data sets.
[0031] Furthermore, using the following formula, the specific steps of training the above weed recognition model include:
[0032] S006, acquiring a plurality of historical harvest images and historical marked images, wherein the historical marked images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; associating the historical harvest images and the historical marked images;
[0033] S007, dividing the plurality of historical acquisition images into a training set and a test set, and using the historical marked images as a verification set;
[0034] S008. Calling a neural network model and configuring parameters of the neural network model;
[0035] Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position;
[0036] S009. Calling a historically marked image from the verification set, the historically marked image is associated with the historical harvest image input into the neural network model, and by matching the historically marked image with the output image, updating the parameters of the neural network model to obtain a trained weed recognition model.
[0037] Obtain historical data: First, collect multiple historical harvest images and corresponding historical marked images. These historical marked images are images after the first weed position and the second weed position are manually marked on the original historical harvest images; ensure that each historical harvest image is correctly associated with its corresponding historical marked image to facilitate comparison of the output result and the actual result; divide the acquired historical harvest images into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate the performance of the model after the training is completed; use the historical marked images as the validation set. The validation set is used to adjust the parameters of the model during the training process to ensure that the model performs well on unseen data; select a suitable neural network model as the basis for weed recognition. This model can be a convolutional neural network (CNN), a recurrent neural network (RNN), or other models suitable for image recognition; configure the parameters of the neural network model, including the number of layers, the number of neurons, the learning rate, the optimizer, etc.; The historical harvest images in the training set are input into the neural network model, and the model will output images with the first weed position and the second weed position marked (the model will output two images in succession, the first image only marks the first weed position, and the second image marks both the first weed position and the second weed position, so the output image at this time is the second image), and the difference between the output image and the true marked image (that is, the historical marked image in the validation set) is calculated. The loss function is usually used to measure this difference, and then the parameters of the model are updated through the back propagation algorithm to reduce the value of the loss function; the above process is repeated until the performance of the model on the validation set reaches the preset standard or the number of training times reaches the upper limit; after the training is completed, the performance of the model is evaluated using the test set, which helps to understand the generalization ability of the model on unseen data; according to the performance on the test set, the parameters of the model can be further adjusted; after sufficient training and evaluation, a trained weed recognition model is obtained.
[0038] A second aspect provides a leafy vegetable harvesting system, the harvesting system adopting the above-mentioned harvesting method;
[0039] The harvesting system includes:
[0040] A mobile carrier, wherein a first positioning module is provided on the mobile carrier, and the first positioning module is used to locate the position of the mobile carrier to obtain a first positioning coordinate;
[0041] A collection module, which is arranged in front of the mobile carrier. The collection module is used to collect the image to be collected in the collection area after receiving the collection task. A second positioning module is arranged on the collection module. The second positioning module is used to locate the position of the collection module and obtain the second positioning coordinates.
[0042] A processing module, the processing module is connected to the acquisition module, the first positioning module and the second positioning module, and the processing module is used to perform the following steps:
[0043] The image to be harvested is input into the trained weed recognition model, and the weed recognition model performs the following steps:
[0044] calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;
[0045] generating and sending a removal instruction according to the first weed position in the first to-be-harvested marked image;
[0046] calling a second weed image set, identifying the second weeds on the first to-be-harvested marked image through the second weed image set, marking the position of the second weeds on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;
[0047] calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;
[0048] Determine a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image;
[0049] Generate harvesting instructions based on the above-mentioned cutting blank area;
[0050] When the first positioning coordinates are moved to the second positioning coordinates, a harvesting instruction is sent;
[0051] The execution module is connected to the processing module. After receiving the clearing instruction, the execution module clears the weeds at the first weed position, and after receiving the harvesting instruction, the execution module harvests the leafy vegetables in the harvesting area.
[0052] The above-mentioned mobile carrier serves as a mobile platform of the entire harvesting system and is responsible for moving in the farmland to perform the harvesting task; the acquisition module is arranged in front of the mobile carrier to ensure that the image to be harvested in front can be acquired in real time during the movement; the execution module is connected to the processing module and is responsible for executing the instructions from the processing module. The positioning modules are respectively arranged on the mobile carrier and the acquisition module to determine the current positions of the mobile carrier and the acquisition module; after generating the clearing instruction, the processing module immediately sends the clearing instruction to the execution module to clear the marked first weeds; after generating the harvesting instruction, it is temporarily not sent to the execution module. When the first positioning coordinates coincide with the second positioning coordinates or meet specific conditions, the harvesting instruction is sent to the execution module.
[0053] Furthermore, the execution module includes a pulling mechanism, a cutting mechanism and a transport mechanism, the pulling mechanism is arranged in front of the mobile carrier, and the collection module is connected to the pulling mechanism;
[0054] One end of the transport mechanism is connected to the front of the mobile carrier, and the other end extends to a side away from the mobile carrier and tilts downward along the extension direction;
[0055] The cutting mechanism is arranged at one end of the transport mechanism away from the mobile carrier, and is used for cutting leafy vegetables.
[0056] The above-mentioned pulling mechanism is arranged in front of the mobile carrier, as the front end part of the execution module, directly facing the farmland area to be harvested; after receiving the clearing instruction, the pulling mechanism will start and remove the marked first weed (which can be achieved by a robotic arm or a similar device, which can accurately locate and pull out the weeds while avoiding damage to the surrounding leafy vegetables); the collection module is connected to the pulling mechanism, and while pulling out the weeds, the collection module can continue to collect the images to be harvested in front, providing data support for subsequent leafy vegetable identification; the main function of the transportation mechanism is to transport the cut leafy vegetables from the harvesting area to the designated collection point; the cutting mechanism is arranged at the end of the transportation mechanism away from the mobile carrier, and the cutting mechanism and the transportation mechanism work closely together to ensure that the cut leafy vegetables can fall directly on the transportation mechanism and then be transported to the collection point along the inclined track.
[0057] Furthermore, the cutting mechanism comprises a support shaft, and connecting pieces are provided at both ends of the support shaft, and the connecting pieces are connected to the transport mechanism;
[0058] A plurality of blades are arranged on the support shaft, and the blades reciprocate along the axial direction of the support shaft, and the movement directions of adjacent blades are opposite;
[0059] The above processing module is used to configure the blade code. Each blade has a unique code, and the corresponding code is determined by cutting the blank area.
[0060] When the cutting mechanism is used to execute a harvesting instruction, the blade corresponding to the code stops moving toward the cutting blank area.
[0061] The above-mentioned support shaft serves as the core component of the cutting mechanism. The support shaft not only provides stable support for the blades, but also ensures that they can reciprocate along the axial direction. The connecting pieces are located at both ends of the support shaft, which are used to connect the support shaft and the transport mechanism, ensuring that the cutting mechanism can be firmly installed at the end of the transport mechanism and move with the movement of the mobile carrier. Multiple blades are evenly distributed along the support shaft, and each blade can independently reciprocate along the axial direction of the support shaft, so that the cutting mechanism can handle multiple cutting tasks at the same time, significantly improving the cutting efficiency. The adjacent blades move in opposite directions. When two adjacent blades move relative to each other, the space between them will gradually shrink, thereby achieving the cutting of leafy vegetables. When two adjacent blades move back to back, the space between them will gradually increase, forming a blank area. The processing module configures a unique code for each blade. These codes are not only used to identify and manage the blades, but also can determine the blade that needs to stop moving according to the information of the cutting blank area when executing the harvesting instruction; when the processing module issues a harvesting instruction, it will calculate the code of the blade that needs to stop moving according to the information of the cutting blank area. Then, a stop signal is sent to the corresponding blade through the control system to ensure that weeds in the blank area will not be cut during the cutting process.
[0062] Furthermore, the removal mechanism comprises a fixing member and a removal member, one end of the fixing member is connected to the movable carrier, and the other end of the fixing member is provided with a connecting rod, and the connecting rod is provided with a slide rail;
[0063] One end of the plucking member is connected to the slide rail, and the plucking member moves along the slide rail; the other end of the plucking member is provided with a clamping portion, and the clamping portion is used to clamp the first weeds.
[0064] The above-mentioned fixing member serves as a supporting member of the plucking mechanism, and one end of the fixing member is connected to the mobile carrier, thereby ensuring the stability and reliability of the plucking mechanism; a connecting rod is provided at the other end of the fixing member, which not only connects the fixing member and the plucking member, but also provides a moving path for the plucking member; a slide rail is provided on the connecting rod, which provides a guide for the movement of the plucking member, and the plucking member moves on the slide rail to achieve the plucking of weeds; one end of the plucking member is connected to the slide rail, and can move smoothly along the slide rail, thereby ensuring that the plucking member can accurately locate the position of the weeds; a clamping part is provided on the plucking member, and the clamping part usually adopts a mechanical clamp or a similar device, which can clamp the weeds to ensure that they will not fall off during the plucking process.
[0065] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0066] A phased recognition strategy is adopted. Obvious weeds are first identified through the first weed image set, and then further identified through the second weed image set. The first phase of recognition can quickly filter out most of the non-leafy areas, reducing the amount of image data to be processed later; phased recognition allows the use of more targeted models and data sets at each stage, thereby improving the accuracy of recognition; each image set is optimized for specific targets (such as weeds or leafy vegetables), making the model more efficient in identifying these targets; after weeds and leafy vegetables are identified at each stage, clearing instructions and harvesting instructions are generated respectively. By clearly specifying the locations of weeds to be cleared and the areas of leafy vegetables that do not need to be harvested, precise control of the harvesting position is achieved, reducing the amount of weeds mixed into leafy vegetables after harvesting; the use of phased recognition and targeted image sets reduces unnecessary calculations, indirectly reduces the complexity of the algorithm, thereby improving the overall recognition speed and meeting the real-time requirements of mechanized harvesting. In addition, in actual use, there is a time difference between the execution of the first and second phase instructions, which is sufficient to support the delay in weed recognition, reduce the risk of misidentification, and provide strong support for agricultural automation and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0068] Figure 1 It is the main flow chart of the harvesting method;
[0069] Figure 2 It is the connection diagram of the harvesting system;
[0070] Figure 3 It is the overall plan view of the harvesting system;
[0071] Figure 4 It is a structural diagram of the cutting mechanism.
[0072] Marks and corresponding parts names in the attached drawings:
[0073] 1. Mobile carrier; 2. Collection module; 3. Execution module; 31. Transport mechanism; 32. Cutting mechanism; 321. Support shaft; 322. Connecting piece; 323. Blade; 33. Extraction mechanism; 331. Fixing piece; 332. Connecting rod; 333. Extraction piece; 334. Clamping part; 4. Processing module; 5. First positioning module; 6. Second positioning module. DETAILED DESCRIPTION
[0074] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0075] First embodiment:
[0076] Combination Figure 1 , a leafy vegetable harvesting method based on deep learning, comprising the following steps:
[0077] S100, after receiving the harvesting task, collecting the image to be harvested in the harvesting area;
[0078] S200, inputting the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps:
[0079] S210, calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;
[0080] generating a removal instruction according to the first weed position in the first to-be-harvested marked image;
[0081] S220, calling a second weed image set, identifying a second weed on the first to-be-harvested marked image through the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;
[0082] S230, calling a leafy vegetable image set, identifying leafy vegetables on the second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;
[0083] S240, determining a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image;
[0084] According to the above-mentioned cutting blank area, a harvesting instruction is generated.
[0085] A phased recognition strategy is adopted. Obvious weeds are first identified through the first weed image set, and then further identified through the second weed image set. The first phase of recognition can quickly filter out most of the non-leafy areas, reducing the amount of image data to be processed later; phased recognition allows the use of more targeted models and data sets at each stage, thereby improving the accuracy of recognition; each image set is optimized for specific targets (such as weeds or leafy vegetables), making the model more efficient in identifying these targets; after weeds and leafy vegetables are identified at each stage, clearing instructions and harvesting instructions are generated respectively. By clearly specifying the locations of weeds to be cleared and the areas of leafy vegetables that do not need to be harvested, precise control of the harvesting position is achieved, reducing the amount of weeds mixed into leafy vegetables after harvesting; the use of phased recognition and targeted image sets reduces unnecessary calculations, indirectly reduces the complexity of the algorithm, thereby improving the overall recognition speed and meeting the real-time requirements of mechanized harvesting. In addition, in actual use, there is a time difference between the execution of the first and second phase instructions, which is sufficient to support the delay in weed recognition, reduce the risk of misidentification, and provide strong support for agricultural automation and intelligence.
[0086] Second embodiment:
[0087] On the basis of the first embodiment, before training the above weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include:
[0088] S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables;
[0089] S002, converting the leafy vegetable images and the weed images into feature vectors to obtain leafy vegetable feature vectors and weed feature vectors;
[0090] S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector;
[0091] S004, determining the similarity between the leafy vegetable image and the weed image through the leafy vegetable unit vector and the weed unit vector;
[0092] S005, calling the similarity threshold, and comparing the above similarity with the similarity threshold;
[0093] When the similarity is less than the similarity threshold, the weed image is classified as the first weed image;
[0094] When the similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.
[0095] By collecting images of weeds mixed with leafy vegetables, it is possible to ensure that the dataset contains various possible weed types, growth stages, and relative positional relationships with leafy vegetables, thereby improving the generalization ability of the model; a dataset containing rich weed samples helps the model learn to distinguish the features of weeds and leafy vegetables, reducing the misrecognition rate in practical applications; converting images into feature vectors allows the model to extract useful information from the original image data; normalizing the feature vectors to ensure that all feature vectors are on the same scale. The normalized feature vectors are more efficient in calculation because they are more uniform in numerical value, reducing errors and unnecessary calculations in the calculation process; by calculating the similarity between the leaf unit vector and the weed unit vector, the degree of similarity between them is quantified, and the weed images are classified by the similarity threshold.
[0096] Dividing the weed images into the first weed image set and the second weed image set helps the model better learn the characteristics of different weed types during training. Although this step itself does not directly reduce the complexity of the algorithm, by optimizing the dataset, the model is more efficient during training, thereby indirectly improving the recognition speed and meeting the real-time requirements of mechanized harvesting. Models trained using carefully constructed datasets have stronger generalization capabilities in practical applications and can better adapt to various complex field environments.
[0097] In a specific embodiment, the similarity between the leafy vegetable image and the weed image is determined using the following formula:
[0098] ;
[0099] in, Represents similarity, with a value range of [-1, 1], 1 represents complete similarity, -1 represents complete opposite, and 0 represents no correlation; Indicates The leaf menu bit vector of dimensions; Indicates The weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.
[0100] The above formula calculates the sum of the dot products of the leaf unit vector and the weed unit vector in each dimension and divides it by the product of their respective moduli to obtain the cosine similarity between two vectors. Cosine similarity measures the degree of similarity between two vectors in direction, not their size or length. Cosine similarity only focuses on the direction of the vector, not its size. Since unit vectors are used, normalization is implicit, which helps to reduce calculation errors caused by different scales of feature vectors. The calculation of cosine similarity is relatively simple and only involves basic vector operations, so it has high computational efficiency when processing large-scale data sets.
[0101] An application scenario for reference, assuming that the leafy vegetable image and the weed image are converted into feature vectors, and the leafy vegetable feature vector obtained after normalization is and the weed unit vector Contains 4 dimensions, such as , ,in, , , and Represents the leaf menu bit vectors of the 1st, 2nd, 3rd and 4th dimensions respectively , , , and Represents the weed unit vectors of the 1st, 2nd, 3rd and 4th dimensions respectively ; First calculate the leaf menu bit vector and the weed unit vector The dot product of , and then calculate the leaf menu bit vector and the weed unit vector Length of mold and , and finally calculate the cosine similarity ; When the cosine similarity is less than the similarity threshold, the weed image is classified as the first weed image; when the above similarity is equal to or greater than the similarity threshold, the weed image is classified as the second weed image.
[0102] Third embodiment:
[0103] Based on any of the above embodiments, the specific steps of training the above weed recognition model using the following formula include:
[0104] S006, acquiring a plurality of historical harvest images and historical marked images, wherein the historical marked images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; associating the historical harvest images and the historical marked images;
[0105] S007, dividing the plurality of historical acquisition images into a training set and a test set, and using the historical marked images as a verification set;
[0106] S008. Calling a neural network model and configuring parameters of the neural network model;
[0107] Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position;
[0108] S009. Calling a historically marked image from the verification set, the historically marked image is associated with the historical harvest image input into the neural network model, and by matching the historically marked image with the output image, updating the parameters of the neural network model to obtain a trained weed recognition model.
[0109] Obtain historical data: First, collect multiple historical harvest images and corresponding historical marked images. These historical marked images are images after the first weed position and the second weed position are manually marked on the original historical harvest images; ensure that each historical harvest image is correctly associated with its corresponding historical marked image to facilitate comparison of the output result and the actual result; divide the acquired historical harvest images into a training set and a test set. The training set is used to train the model, while the test set is used to evaluate the performance of the model after the training is completed; use the historical marked images as the validation set. The validation set is used to adjust the parameters of the model during the training process to ensure that the model performs well on unseen data; select a suitable neural network model as the basis for weed recognition. This model can be a convolutional neural network (CNN), a recurrent neural network (RNN), or other models suitable for image recognition; configure the parameters of the neural network model, including the number of layers, the number of neurons, the learning rate, the optimizer, etc.; The historical harvest images in the training set are input into the neural network model, and the model will output images with the first weed position and the second weed position marked (the model will output two images in succession, the first image only marks the first weed position, and the second image marks both the first weed position and the second weed position, so the output image at this time is the second image), and the difference between the output image and the true marked image (that is, the historical marked image in the validation set) is calculated. The loss function is usually used to measure this difference, and then the parameters of the model are updated through the back propagation algorithm to reduce the value of the loss function; the above process is repeated until the performance of the model on the validation set reaches the preset standard or the number of training times reaches the upper limit; after the training is completed, the performance of the model is evaluated using the test set, which helps to understand the generalization ability of the model on unseen data; according to the performance on the test set, the parameters of the model can be further adjusted; after sufficient training and evaluation, a trained weed recognition model is obtained.
[0110] Fourth embodiment:
[0111] Combination Figure 2 , a leafy vegetable harvesting system, the harvesting system adopts the above-mentioned harvesting method;
[0112] The harvesting system includes:
[0113] A mobile carrier 1, wherein the mobile carrier 1 is provided with a first positioning module 5, and the first positioning module 5 is used to locate the position of the mobile carrier 1 to obtain a first positioning coordinate;
[0114] A collection module 2, which is arranged in front of the mobile carrier 1. The collection module 2 is used to collect the image to be harvested in the harvesting area after receiving the harvesting task; a second positioning module 6 is arranged on the collection module 2, and the second positioning module 6 is used to locate the position of the collection module 2 to obtain a second positioning coordinate;
[0115] The processing module 4 is connected to the acquisition module 2, the first positioning module 5 and the second positioning module 6. The processing module 4 is used to perform the following steps:
[0116] The image to be harvested is input into the trained weed recognition model, and the weed recognition model performs the following steps:
[0117] calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image;
[0118] generating and sending a removal instruction according to the first weed position in the first to-be-harvested marked image;
[0119] calling a second weed image set, identifying the second weeds on the first to-be-harvested marked image through the second weed image set, marking the position of the second weeds on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image;
[0120] calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions;
[0121] Determine a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image;
[0122] Generate harvesting instructions based on the above-mentioned cutting blank area;
[0123] When the first positioning coordinates are moved to the second positioning coordinates, a harvesting instruction is sent;
[0124] The execution module 3 is connected to the processing module 4. After receiving the clearing instruction, the execution module 3 clears the weeds at the first weed position, and after receiving the harvesting instruction, the execution module 3 harvests the leafy vegetables in the harvesting area.
[0125] The mobile carrier 1 is a mobile platform of the entire harvesting system, and is responsible for moving in the farmland to perform harvesting tasks. The acquisition module 2 is arranged in front of the mobile carrier 1 to ensure that the image to be harvested in front can be acquired in real time during the movement. The execution module 3 is connected to the processing module 4, and is responsible for executing the instructions from the processing module 4. Positioning modules are respectively arranged on the mobile carrier 1 and the acquisition module 2 to determine the current positions of the mobile carrier 1 and the acquisition module 2. After generating a clearing instruction, the processing module 4 immediately sends the clearing instruction to the execution module 3 to clear the marked first weeds. After generating a harvesting instruction, it is temporarily not sent to the execution module 3. When the first positioning coordinates coincide with the second positioning coordinates or meet specific conditions, the harvesting instruction is sent to the execution module 3.
[0126] Fifth embodiment:
[0127] Based on the fourth embodiment, Figure 3 The execution module 3 includes a pulling mechanism 33, a cutting mechanism 32 and a transport mechanism 31. The pulling mechanism 33 is arranged in front of the mobile carrier 1, and the collection module 2 is connected to the pulling mechanism 33;
[0128] One end of the transport mechanism 31 is connected to the front of the mobile carrier 1, and the other end extends to a side away from the mobile carrier 1 and tilts downward along the extension direction;
[0129] The cutting mechanism 32 is arranged at one end of the transport mechanism 31 away from the mobile carrier 1, and is used for cutting leafy vegetables; the cutting mechanism 32 and the transport mechanism 31 operate synchronously; the pulling mechanism 33 extends out of the end of the cutting mechanism 32 and is located above the cutting mechanism 32.
[0130] The above-mentioned pulling mechanism 33 is arranged in front of the mobile carrier 1, as the front end part of the execution module 3, directly facing the farmland area to be harvested; after receiving the clearing instruction, the pulling mechanism 33 will start and remove the marked first weeds (which can be achieved by a robotic arm or a similar device, which can accurately locate and pull out the weeds while avoiding damage to the surrounding leafy vegetables); the collection module 2 is connected to the pulling mechanism 33. While pulling out the weeds, the collection module 2 can continue to collect the images to be harvested in front, providing data support for subsequent leafy vegetable identification; the main function of the transportation mechanism 31 is to transport the cut leafy vegetables from the harvesting area to the designated collection point; the cutting mechanism 32 is arranged at one end of the transportation mechanism 31 away from the mobile carrier 1, and the cutting mechanism 32 cooperates closely with the transportation mechanism 31 to ensure that the cut leafy vegetables can fall directly on the transportation mechanism 31 and then be transported to the collection point along the inclined track.
[0131] The workflow of the harvesting system:
[0132] (1) The mobile carrier 1 moves in the farmland, and the weeding mechanism 33 removes the weeds according to the removal instruction, while the acquisition module 2 continues to acquire the image to be harvested in front;
[0133] (2) The processing module 4 receives the image to be harvested and calls the trained weed recognition model to identify weeds and leafy vegetables; based on the recognition results, it generates a removal instruction and a harvesting instruction and sends them to the execution module 3;
[0134] (3) The weeding mechanism 33 removes the weeds according to the clearing instruction, while the cutting mechanism 32 cuts the leafy vegetables in the designated cutting blank area according to the harvesting instruction;
[0135] (4) The cut leafy vegetables fall onto the transport mechanism 31 and are then transported to a collection point along an inclined track.
[0136] Sixth embodiment:
[0137] Based on the fifth embodiment, Figure 4 The cutting mechanism 32 includes a support shaft 321, and both ends of the support shaft 321 are provided with connecting members 322, and the connecting members 322 are connected to the transport mechanism 31;
[0138] The support shaft 321 is provided with a plurality of blades 323, each blade 323 can be connected to a driver, the driver can be nested inside the support shaft 321, the driver is connected to the processing module 4, and is used to drive the blade 323 to move according to the harvesting instruction; the driver controls the blade 323 to reciprocate along the axial direction of the support shaft 321, and the movement directions of adjacent blades 323 are opposite;
[0139] The processing module 4 is used to configure the blade 323 code (i.e., the driver code). Each blade 323 has a unique code, and the corresponding code is determined by cutting the blank area.
[0140] When the cutting mechanism 32 is used to execute a harvesting instruction, the blade 323 corresponding to the code stops moving toward the cutting blank area.
[0141] The support shaft 321 is a core component of the cutting mechanism 32. The support shaft 321 not only provides stable support for the blades 323, but also ensures that they can reciprocate along the axial direction. The connecting pieces 322 are located at both ends of the support shaft 321, and are used to connect the support shaft 321 and the transport mechanism 31, so as to ensure that the cutting mechanism 32 can be firmly installed at the end of the transport mechanism 31 and move with the movement of the mobile carrier 1. The multiple blades 323 are evenly distributed along the support shaft 321, and each blade 323 can independently reciprocate along the axial direction of the support shaft 321, so that the cutting mechanism 32 can handle multiple cutting tasks at the same time, which significantly improves the cutting efficiency. The adjacent blades 323 move in opposite directions. When two adjacent blades 323 move relative to each other, the space between them will gradually shrink, thereby achieving the cutting of leafy vegetables; when two adjacent blades 323 move in opposite directions, the space between them will gradually increase, forming a blank area; the processing module 4 configures a unique code for each blade 323, which is not only used to identify and manage the blades 323, but also can determine the blade 323 that needs to stop moving according to the information of the blank area when executing the harvesting instruction; when the processing module 4 issues a harvesting instruction, it will calculate the code of the blade 323 that needs to stop moving according to the information of the blank area. Then, a stop signal is sent to the corresponding blade 323 through the control system to ensure that weeds in the blank area will not be cut during the cutting process.
[0142] The working process of the processing module 4 and the cutting mechanism 32 is as follows:
[0143] The processing module 4 determines the code of the blade 323 that needs to stop moving according to the cutting blank area and the configured blade 323 code;
[0144] The cutting mechanism 32 performs the cutting operation according to the harvesting instruction, and the blade 323 corresponding to the code stops moving when reaching the cutting blank area, thereby reducing the cutting of the second weeds.
[0145] Through the design of the cutting mechanism 32, the leafy vegetable harvesting system can not only efficiently remove weeds and collect leafy vegetables, but also control the movement of the blade 323 according to the actual cutting blank area information to reduce the cutting of the second weeds.
[0146] Seventh embodiment:
[0147] Based on the fifth embodiment, Figure 3 The removal mechanism 33 includes a fixing member 331 and a removal member 333. One end of the fixing member 331 is connected to the mobile carrier 1. The other end of the fixing member 331 is provided with a connecting rod 332. The connecting rod 332 is provided with a slide rail (not shown). Figure 3 (drawn in);
[0148] One end of the plucking member 333 is connected to the slide rail, and the plucking member 333 moves along the slide rail; the other end of the plucking member 333 is provided with a clamping portion 334, and the clamping portion 334 is used to clamp the first weed.
[0149] The fixing member 331 is used as a supporting member of the plucking mechanism 33. One end of the fixing member 331 is connected to the mobile carrier 1, which ensures the stability and reliability of the plucking mechanism 33. The other end of the fixing member 331 is provided with a connecting rod 332, which not only connects the fixing member 331 and the plucking member 333, but also provides a moving path for the plucking member 333. The slide rail is provided on the connecting rod 332, which provides a guide for the movement of the plucking member 333. The plucking member 333 moves on the slide rail to achieve the plucking of weeds. One end of the member 333 is connected to the slide rail and can move smoothly along the slide rail, ensuring that the plucking member 333 can accurately locate the weeds; the plucking member 333 is provided with a clamping portion 334, which usually adopts a mechanical clamp or a similar device, and is driven by a driver to achieve clamping. There is no specific limitation on the clamping portion 334, the driver and the slide rail, which is only used to realize that after the clamping portion 334 clamps the weeds, the plucking member 333 moves along the slide rail, pulls the weeds to pull them out, and throws the weeds out of the harvesting area.
[0150] The working process of the extraction mechanism 33:
[0151] The processing module 4 can determine the actual coordinates of the second weeds according to the position of the second weeds marked on the image to be harvested and the second positioning coordinates, and determine the moving path of the plucking member 333 according to the actual coordinates of the second weeds and the current coordinates of the plucking member 333; the plucking member 333 moves to the designated position along the connecting rod 332 under the guidance of the slide rail; the clamping part 334 closes and clamps the weeds after reaching the designated position; after clamping the weeds, the plucking member 333 moves along the slide rail to pull the weeds out of the soil; after pulling out the weeds, the clamping part 334 opens and releases the weeds, and then the plucking member 333 returns to the initial position to prepare for the next plucking operation; it is worth noting that the moving speed of the mobile carrier 1 is usually slow during harvesting, and there is no need to consider the problem that the plucking member 333 cannot keep up with the moving speed when the moving speed is too fast.
[0152] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A leafy vegetable harvesting method based on deep learning, characterized in that: The following steps are involved: S100, after receiving the harvesting task, collecting the image to be harvested in the harvesting area; S200: Input the image to be harvested into a trained weed recognition model, and the weed recognition model performs the following steps: S210, calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image; generating a removal instruction according to a first weed position in the first to-be-harvested marked image; S220, calling a second weed image set, identifying a second weed on the first to-be-harvested marked image through the second weed image set, marking a position of the second weed on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image; S230, calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions; S240, determining a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image; A harvesting instruction is generated according to the cutting blank area.
2. The leafy vegetable harvesting method according to claim 1, characterized in that: Before training the weed recognition model, it is necessary to construct a first weed image set and a second weed image set. The specific steps include: S001, collecting images of leafy vegetables and images of weeds mixed with the corresponding leafy vegetables; S002, converting the leafy vegetable image and the weed image into feature vectors to obtain a leafy vegetable feature vector and a weed feature vector; S003, normalizing the leafy vegetable feature vector and the weed feature vector to obtain a leafy vegetable unit vector and a weed unit vector; S004, determining the similarity between the leafy vegetable image and the weed image through the leafy vegetable unit vector and the weed unit vector; S005, calling a similarity threshold, and comparing the similarity with the similarity threshold; When the similarity is less than the similarity threshold, classifying the weed image as a first weed image; When the similarity is equal to or greater than the similarity threshold, the weed image is classified as a second weed image.
3. The leafy vegetable harvesting method according to claim 2, characterized in that: Use the following formula to determine the similarity between leafy vegetable images and weed images: ; in, Indicates similarity; Indicates The leaf menu bit vector of dimensions; Indicates The weed unit vector of dimensions; Represents the total dimension of the leafy vegetable feature vector and the weed feature vector.
4. The leafy vegetable harvesting method according to claim 1, characterized in that: Using the following formula, the specific steps of training the weed recognition model include: S006, acquiring a plurality of historical harvest images and historical marking images, wherein the historical marking images are images obtained by marking the first weed position and the second weed position on the corresponding historical harvest images; and associating the historical harvest images with the historical marking images; S007, dividing the plurality of historical acquisition images obtained into a training set and a test set, and using the obtained historical marked images as a verification set; S008. Calling a neural network model and configuring parameters of the neural network model; Inputting the historical harvest images in the training set into the neural network model to obtain an output image, wherein the output image is an image after marking the first weed position and the second weed position; S009, calling a historically marked image from the verification set, associating the historically marked image with the historical harvest image input into the neural network model, and updating the parameters of the neural network model by matching the historically marked image with the output image to obtain a trained weed recognition model.
5. A leafy vegetable harvesting system, characterized in that: The harvesting system adopts the harvesting method according to any one of claims 1 to 4; The harvesting system includes: A mobile carrier (1), wherein a first positioning module (5) is provided on the mobile carrier (1), and the first positioning module (5) is used to locate the position of the mobile carrier (1) to obtain a first positioning coordinate; A collection module (2), the collection module (2) being arranged in front of the mobile carrier (1), the collection module (2) being used to collect the image to be collected in the collection area after receiving the collection task; a second positioning module (6) being arranged on the collection module (2), the second positioning module (6) being used to locate the position of the collection module (2) to obtain a second positioning coordinate; A processing module (4), the processing module (4) is connected to the acquisition module (2), the first positioning module (5) and the second positioning module (6), and the processing module (4) is used to perform the following steps: The image to be harvested is input into a trained weed recognition model, and the weed recognition model performs the following steps: calling a first weed image set, identifying a first weed on the to-be-harvested image through the first weed image set, marking a position of the first weed on the to-be-harvested image, and obtaining a first to-be-harvested marked image; generating and sending a removal instruction according to a first weed position in the first to-be-harvested marked image; calling a second weed image set, identifying second weeds on the first to-be-harvested marked image through the second weed image set, marking a position of the second weeds on the first to-be-harvested marked image, and obtaining a second to-be-harvested marked image; calling a leafy vegetable image set, identifying leafy vegetables on a second to-be-harvested marked image through the leafy vegetable image set, extracting leafy vegetable positions adjacent to the second weed positions from the second to-be-harvested marked image, and obtaining leafy vegetable extraction positions; Determine a blank area for cutting according to the second weed position and the leafy vegetable extraction position in the second to-be-harvested marked image; Generate a harvesting instruction according to the cutting blank area; When the first positioning coordinate moves to the second positioning coordinate, sending a harvesting instruction; An execution module (3) is connected to the processing module (4), and after receiving a clearing instruction, the execution module (3) clears the weeds at the first weed position, and after receiving a harvesting instruction, the execution module (3) harvests the leafy vegetables in the harvesting area.
6. The leafy vegetable harvesting system according to claim 5, characterized in that: The execution module (3) comprises a plucking mechanism (33), a cutting mechanism (32) and a transport mechanism (31); the plucking mechanism (33) is arranged directly in front of the mobile carrier (1); and the collection module (2) is connected to the plucking mechanism (33); One end of the transport mechanism (31) is connected to the front of the mobile carrier (1), and the other end extends to a side away from the mobile carrier (1) and is inclined downward along the extension direction; The cutting mechanism (32) is arranged at one end of the transport mechanism (31) away from the mobile carrier (1), and the cutting mechanism (32) is used for cutting leafy vegetables.
7. The leafy vegetable harvesting system according to claim 6, characterized in that: The cutting mechanism (32) comprises a supporting shaft (321), and connecting pieces (322) are provided at both ends of the supporting shaft (321), and the connecting pieces (322) are connected to the transport mechanism (31); A plurality of blades (323) are arranged on the support shaft (321), and the blades (323) reciprocate along the axial direction of the support shaft (321), and adjacent blades (323) move in opposite directions; The processing module (4) is used to configure the blade (323) code, each blade (323) has a unique code, and the corresponding code is determined by cutting the blank area; When the cutting mechanism (32) is used to execute a harvesting instruction, the blade (323) corresponding to the code stops moving toward the cutting blank area.
8. The leafy vegetable harvesting system according to claim 6, characterized in that: The pulling mechanism (33) comprises a fixing member (331) and a pulling member (333), one end of the fixing member (331) is connected to the movable carrier (1), and the other end of the fixing member (331) is provided with a connecting rod (332), and a slide rail is provided on the connecting rod (332); One end of the plucking member (333) is connected to the slide rail, and the plucking member (333) moves along the slide rail; the other end of the plucking member (333) is provided with a clamping portion (334), and the clamping portion (334) is used to clamp the first weed.
Citation Information
Patent Citations
Weed recognition method and device, computing equipment and storage medium
CN113807130A
Weed detection method and device based on crop growth characteristics
CN114818909A
Paddy field seedling weed stem center distinguishing and positioning method based on two-stage segmentation model
CN117496353A
Automatic weeding method based on deep learning
CN118115879A
Machine learning models for selecting treatments for treating an agricultural field
US20220092705A1
Cited By
Control system and method for lithium battery pruning shears
CN119272033A
Harvesting and collecting device for salvia miltiorrhiza rhizome traditional Chinese medicinal materials in sticky and heavy soil
CN120380923A
