Method and system for identifying crops and weeds based on machine vision

Abstract

The present invention provides a method and readable medium for identifying crops and weeds based on machine vision. The method includes capturing images of farmland and inputting them into an object detection model for identification and classification into at least one crop and at least one weed. Subsequently, the center point calculation of crops and weeds is performed to obtain farmland images with center points, thereby identifying and defining at least one retention area and at least one removal area. The imaging device based on the movement of the vehicle along the direction of the farmland, captures multiple images of the farmland. These images undergo 3D spatial transformation to construct a 3D spatial distribution map of the crops in the farmland.

Description

[Technical Field]

[0001] This invention relates to a seedling thinning and weeding method for smart agriculture, and more particularly to a method and system for judging crops and weeds based on machine vision. [Previous Technology]

[0002] Taiwan faces problems such as an aging population and a declining birth rate, resulting in fewer and fewer people working in agriculture. Furthermore, the natural environment has increased the cost of agriculture in Taiwan. In response to this problem, smart agriculture has become the main development direction for agriculture in Taiwan. Smart agriculture uses big data analysis or artificial intelligence to assist farmers in managing crop growth in an automated way.

[0003] Furthermore, short-cycle crops are among the mainstream crops grown in Taiwan, with a growth cycle of 4 to 5 weeks. To achieve rapid sowing, broadcast sowing is used. However, this method requires thinning the seedlings 2 to 3 weeks after sowing to maintain an appropriate distance between seedlings and prevent overcrowding or uneven nutrient distribution. However, judging weeds and thinning methods relies heavily on farmers' experience and subjective judgment, requiring a large amount of manpower for weeding and thinning, and is difficult to automate. [Summary of the Invention]

[0004] In view of this, the present invention discloses a method and system for judging crops and weeds based on machine vision. The method uses machine vision to identify and distinguish crops and weeds, thereby removing overly dense crops to achieve the effect of thinning seedlings. After thinning seedlings and removing weeds in the farmland, a three-dimensional spatial distribution map of crops in the farmland is established, which can more effectively improve the efficiency of crop management.

[0005] This invention provides a method for identifying crops and weeds based on machine vision, comprising the following steps: capturing an image; a vehicle having a shooting device moves along the direction of a farmland, and the shooting device captures an image of the farmland; performing crop identification; inputting the farmland image into an object detection model; after identification by the object detection model, defining and classifying at least one object in the farmland image as at least one crop and at least one weed, to form an image containing the crops and weeds. The system identifies the image; determines the removal area and the retention area; calculates a center point based on the identified image containing the crops and weeds to obtain a farmland image with a center point; and defines at least one removal area and at least one retention area based on the farmland image with the center point; if it is a removal area, an operating arm of the vehicle performs a removal action to remove the crops and weeds in the removal area; and if it is a retention area, the operating arm of the vehicle does not perform the removal action in the retention areas.

[0006] In some embodiments, a three-dimensional spatial map of farmland and crops is constructed. The imaging device moves along the direction of the farmland according to the vehicle to obtain a plurality of images of farmland containing the reserved areas. The plurality of images of farmland containing the reserved areas are subjected to a three-dimensional spatial transformation process to construct a three-dimensional spatial map of farmland and crops.

[0007] In some embodiments, the steps of agricultural object recognition further include the following steps: identifying and classifying objects, using the object detection model to identify and classify the objects in the agricultural land image, and marking each object with an identification bounding box, and forming an identified object within the identification bounding box; contour segmentation, performing contour detection on the identified object to obtain an object contour; and center positioning, obtaining a center point based on the centroid of the object contour to form an identification box image with a center point.

[0008] In some embodiments, the method for training the object detection model further includes the following steps: labeling and classifying images, labeling the objects in the farmland image, and defining the categories as the crops and the weeds respectively; expanding the farmland image, performing image augmentation on the labeled farmland image to obtain multiple labeled farmland images; and outputting the object detection model, inputting the labeled farmland images into a machine learning model for training, and outputting the object detection model.

[0009] In some embodiments, the center point calculation further includes setting a distance value and setting a center distance between the crops, wherein the distance between the crops is greater than or equal to the center distance and is displayed as the retained areas, and the distance between the crops is less than the center distance and is displayed as the removed areas.

[0010] In some embodiments, the method further includes the following steps: obtaining pixel coordinates, wherein the identification bounding box of the identification box image of the center point has a center point position and a depth value, and obtaining a corresponding pixel coordinate; converting three-dimensional spatial coordinates, obtaining a three-dimensional spatial coordinate by performing a three-dimensional coordinate formula conversion based on the pixel coordinates and the depth value; and converting the three-dimensional coordinates of the vehicle, obtaining the three-dimensional coordinates of the vehicle based on the three-dimensional spatial coordinates and the external parameter matrix transformation formula.

[0011] In some embodiments, the three-dimensional farmland and crop distribution map can be projected to form a two-dimensional farmland and crop distribution map.

[0012] In some embodiments, the three-dimensional spatial transformation process further includes the following steps: outputting data, wherein the multiple images of farmland containing the reserved areas output a timestamp, a world coordinate of the imaging device, and a quaternion of the imaging device, and the multiple images of farmland containing the reserved areas are formed by the imaging device capturing different farmland images in different areas of the farmland as the vehicle moves through the farmland; and converting coordinates, wherein the quaternion of the imaging device is converted according to the timestamp and the world coordinate of the imaging device. The system performs a coordinate transformation to form a coordinate transformation matrix, and outputs a feature coordinate of the preserved areas and the coordinate transformation matrix. It also establishes a three-dimensional map of farmland and crop distribution. If a portion of the complex images containing the farmland of the preserved areas has the corresponding feature coordinate, and another portion of the complex images containing the farmland of the preserved areas also has the corresponding feature coordinate, the complex images containing the farmland of the preserved areas share the common feature coordinate and are superimposed to establish a three-dimensional map of farmland and crop distribution, displaying the crops of the preserved areas.

[0013] The present invention provides a system for judging crops and weeds based on machine vision, comprising: a vehicle movable on a farmland; a camera disposed on the vehicle, for capturing an image of the farmland based on the position of the vehicle on the farmland; an object detection model for identifying at least one object in the farmland image and classifying and defining the objects as at least one crop and at least one weed; a removal judgment module for setting a center distance based on the identified farmland image and the removal judgment module, for judging whether it is a retention area or a removal area based on the center distance, transmitting a coordinate command to an operating arm of the vehicle to perform a removal action, removing the crops and weeds in the removal area; and a coordinate conversion module for converting the coordinates, converting a pixel coordinate of the farmland image into a three-dimensional coordinate of the camera, and then converting the three-dimensional coordinate into a three-dimensional coordinate of the vehicle.

[0014] In some embodiments, the imaging device is a depth camera (RGB-D) used to capture a depth image of the farmland and an RGB image of the farmland.

[0015] In some embodiments, the object detection model is based on a machine learning-based identification model.

[0016] In some embodiments, a map construction module is further included, wherein the shooting device moves along the direction of the farmland according to the vehicle to obtain a plurality of images of the farmland containing the reserved areas, and the plurality of images of the farmland containing the reserved areas have the reserved areas, and a three-dimensional farmland and crop distribution map is constructed through the reserved areas and a coordinate transformation matrix.

[0017] This invention provides a method and system for judging crops and weeds based on machine vision. It can identify crops and weeds in farmland images by using an object detection model. At the same time, it can quickly perform thinning and weeding based on the growth range between crops. After thinning and weeding the farmland, it can construct a three-dimensional map of the distribution of crops in the farmland. Through this map, it can better control the yield of crops and effectively improve the efficiency of agricultural management, so as to achieve the purpose of this invention.

Implementation Method

[0018] The following describes the implementation of the present invention through specific embodiments, providing a more detailed explanation of the invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different embodiments, and various details in this specification can be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0019] Please refer to Figure 1, which is a block diagram of a system for judging crops and weeds based on machine vision according to the present invention.

[0020] The vehicle 10 is movable on the farmland, and the imaging device 20 is set on the vehicle 10. When the vehicle 10 is located in a part of the farmland, the imaging device 20 captures farmland images according to the area. The imaging device 20 is preferably a depth camera (RGB-D) to capture farmland images of depth map and farmland images of RGB map. The imaging device 20 can measure the Z-axis distance between each image point and the imaging device 20, so as to store three-dimensional stereo images.

[0021] The object detection model 30 is used to identify at least one object in a farmland image and classify and define the object as at least one crop 120 and at least one weed 130, wherein the crop 120 is bok choy, spinach, choy sum or other green vegetables, but the present invention is not limited thereto. Since the farmland image shows crop 120 and weed 130, wherein crop 120 is in the seedling stage, the two are not easy to distinguish and identify. In order to better identify crop 120 and weed 130, the object detection model 30 is needed to distinguish them. The object detection model 30 is a machine learning recognition model, such as YOLOv8x, Mask R-CNN or other detection models, but the present invention is not limited thereto.

[0022] The removal judgment module 40, after being identified by the object detection model 30, is used to distinguish between crops 120 and weeds 130. To avoid the dense growth of crops 120, thinning is performed to maintain the distance between them, allowing them sufficient growth space. A center distance D is set at the exact center of each crop 120 and adjacent crops 120 based on their actual growth status. The growth area of ​​this crop 120 is determined to be either the removal area 150 or the retention area 160 based on the center distance D. If the distance is less than the center distance D, it indicates that the crops 120 need to be thinned and is determined to be the removal area 150; conversely, if the distance is greater than or equal to the center distance D, it indicates that the crops 120 are not densely growing and is determined to be the retention area 160. Coordinate commands are transmitted to the operating arm 50 of the carrier 10 to perform the removal action, removing the crops 120 and weeds 130 located in the removal area 150.

[0023] The coordinate conversion module 60 is based on the object detection model 30 and the removal judgment module 40 to identify and judge based on the farmland image. If the operating arm 50 of the vehicle 10 performs a removal action on the crops 120 and weeds 130 in the removal area 150, it is necessary to transmit the three-dimensional coordinates to the vehicle 10 first. Specifically, the pixel coordinates of the farmland image captured by the shooting device 20 are converted into the three-dimensional coordinates of the shooting device, and then the three-dimensional coordinates are converted into the three-dimensional coordinates of the vehicle 10. In this way, the operating arm 50 of the vehicle 10 can be operated to perform the removal action at the specified position by using the coordinate conversion module 60.

[0024] The map construction module 70 uses the shooting device 20 to obtain multiple images 170 of farmland including the reserved area as the vehicle 10 moves along the direction of the farmland. The multiple images 170 of farmland including the reserved area have a reserved area 160. Through the reserved area 160 and the coordinate transformation matrix, a three-dimensional farmland and crop distribution map is constructed. The three-dimensional farmland and crop distribution map can also be transformed into a two-dimensional farmland and crop distribution map. When the vehicle 10 completes the thinning and weeding of some areas of the farmland, it moves to the next farmland to continue the work until the thinning and weeding of all crops 120 are completed, so as to generate a three-dimensional farmland and crop distribution map of the entire farmland area, which is conducive to the subsequent management of the growth of crops 120.

[0025] To further illustrate the present invention, please refer to Figure 2 and the partial flowcharts in Figures 3 to 6. Figure 2 is a flowchart of a method for judging crops and weeds based on machine vision according to the present invention.

[0026] S10 captures images. The vehicle 10 has a shooting device 20. The vehicle 10 moves along the direction of the farmland, and the shooting device 20 captures images of the farmland.

[0027] S20 performs crop identification, inputs farmland image to object detection model 30, and after identification by object detection model 30, defines and classifies at least one crop 120 and at least one weed 130 to form an image with identification of crops and weeds.

[0028] To further illustrate the method of training the object detection model 30, please refer to Figure 3.

[0029] S100 Label and classify the image, label at least one object in the farmland image, and define the categories as crops 120 and weeds 130 respectively. Since the farmland image includes crops 120 and weeds 130, the software defines and labels the object categories as crops 120 and weeds 130 in the farmland image.

[0030] S110 Expanding farmland images involves augmenting the labeled farmland images to obtain multiple labeled farmland images. Image augmentation includes techniques such as image flipping, changing brightness and exposure, and increasing noise to expand the sample size. For example, image augmentation can expand the original dataset from 57 images to 1153 images, thereby enhancing the model's generalization ability.

[0031] S120 outputs an object detection model, inputs labeled farmland images into a machine learning model for training, and outputs an object detection model 30. For example, the labeled data for crops 120 and weeds 130 are 5818 and 12495 respectively, and the dataset is divided into a training set, a validation set, and a test set according to a 7:2:1 ratio, which are then input into the machine learning model for machine learning to output an object detection model 30.

[0032] The process of labeling and classifying images from S100 to outputting the object detection model from S120 can enhance the recognition accuracy of the object detection model 30 by increasing the number of samples and defining other categories.

[0033] As shown in Figure 4, the agricultural identification step in S20 further includes the following steps:

[0034] S130 identifies and classifies objects, using the object detection model 30 to identify objects in the farmland image, and marks them with identification bounding boxes 80, forming identified objects 90 within the identification bounding boxes 80. Specifically, the farmland image is noise-removed, and then input into the object detection model 30 to identify and classify the detected objects in the farmland image as crops 120 and weeds 130, so that they are marked with identification bounding boxes 80, and forming identified objects 90 within the identification bounding boxes 80.

[0035] S140 Contour segmentation: Contour detection is performed on the identified object 90 to obtain the object contour. Specifically, OpenCV is used to segment the identified object 90 to remove unnecessary detection areas. Then, color space conversion is used to convert the color image to a grayscale image. To highlight the contour features of the identified object 90, channel binarization is performed on the grayscale image to convert it to a black and white image. Finally, image denoising is used to remove noise from the black and white image, thus obtaining the object contour for contour recognition.

[0036] S150 Locate the center, obtain the center point based on the centroid of the object's outline, and form an identification box image 110 with the center point. Specifically, based on the object identification and classification in S130, obtain the coordinate position of the center point, and mark the center point to form an identification box image 110 with the center point.

[0037] In S20, agricultural organism identification is performed, and in S130, objects are identified and classified to the positioning center in S150. As shown in Figures 7A and 7B, after identification by the object detection model 30, a bounding box 80 is formed, which then forms the identified object 90. The center point 100 of the centroid of the identified object 90 is obtained through contour segmentation, thus forming an identification box image 110 with a center point. In this way, an identification box image 110 with a center point is generated, which is classified as agricultural organisms 120 and weeds 130, and the center point of the identification box image is marked, so that it has a center point for subsequent processing.

[0038] S30 determines the removal area and the retention area. Based on the identification image of crops and weeds, a center point is calculated to obtain a farmland image 140 with a center point. The farmland image 140 with the center point is defined as at least one removal area 150 and at least one retention area 160. The center point calculation further includes setting a distance value, specifically setting a center distance D between adjacent crops 120. If the distance between crops 120 is greater than or equal to the center distance D, it is displayed as a retention area 160; otherwise, if the distance between crops 120 is less than the center distance D, it is displayed as a removal area 150. The farmland image includes multiple crops 120 and multiple weeds 130, and the sowing distance between crops 120 is inconsistent. To better determine whether thinning is necessary, a fixed center distance D is set between crops 120, thereby determining whether it is a removal area 150 or a retention area 160.

[0039] Specifically, by generating a farmland image 140 with a center point at the positioning center of S150, a center distance D is set between crops 120 as a reserved growth range, and the distance between each crop 120 and its adjacent crops 120 is calculated. If the distance is less than the center distance D, it is displayed as a removal area 150, and if it is greater than or equal to the center distance D, it is displayed as a retention area 160. The calculation method uses the XY axis coordinates on the farmland image, and calculates the distance using four arrangements of the X-axis coordinates from small to large and from large to small. After calculation, the crop growth area with the most retention area 160 is selected as the better crop growth area to avoid over-thinning.

[0040] If S40 is a removal area and S50 is a retention area, the operating arm 50 of the vehicle 10 performs a removal action, removing the crops 120 and weeds 130 in the removal area 150, while the operating arm 50 does not perform a removal action in the retention area 160. The farmland image 140 with the center point obtained in step S30 is used as the coordinate position to enable the operating arm 50 of the vehicle 10 to perform the removal action.

[0041] In step S30, which determines the removal and retention areas, as shown in Figure 8, the object detection model 30 identifies objects in the farmland image and classifies them into crops 120 and weeds 130. To perform thinning to allow crops 120 sufficient space to grow, a center distance D is set at the center point 100 of the crops 120. The distances to adjacent crops 120 are calculated; those greater than or equal to the center distance D are considered retention areas 160, while those less than the center distance D are removal areas 150. This allows the operating arm 50 of the carrier 10 to perform removal actions within the designated areas based on the coordinates. It should be noted that the crop 120 used is bok choy as an example for practical application. However, the actual crop 120 is not limited to this; it can also be bok choy, spinach, celery, or other green vegetables, but the present invention is not limited to this.

[0042] S160 Construct a three-dimensional spatial map of farmland and agricultural products. The imaging device 20 moves along the direction of farmland according to the vehicle 10 to obtain multiple images 170 of farmland including the reserved area. The multiple images 170 of farmland including the reserved area are processed by three-dimensional spatial transformation to construct a three-dimensional spatial map of farmland and agricultural products.

[0043] As shown in Figure 5, S160 constructing a three-dimensional spatial map of farmland and agricultural product distribution further includes the following steps:

[0044] S170 output data, including multiple images of farmland in the reserved area. The output of S170 includes timestamps, world coordinates of the shooting device, and quaternions of the shooting device. After the vehicle 10 completes weeding and seedling thinning in a part of the farmland, it moves to an adjacent area and outputs a text file of the trajectory of the vehicle 10 in this area, a text file of the reserved area 160, and map point cloud data. The text file includes timestamps, world coordinates of the shooting device, and quaternions of the shooting device. The quaternions represent the translation and rotation relationship of the shooting device 20 relative to three-dimensional space.

[0045] S180 Coordinate Transformation: The quaternion of the shooting device is transformed into a coordinate transformation matrix based on the timestamp and the world coordinates of the shooting device. Region 160 and the coordinate transformation matrix are retained, and the characteristic coordinates of the retained region 160 are output. The quaternion of the shooting device is () and the coordinate transformation matrix is ​​transformed as shown in Equation (1). Equation (1) is, R is the rotation matrix, T is the translation vector, and , and are the displacements along the X, Y and Z axes, respectively. The extrinsic parameter system is used to establish the coordinates of the shooting device and the coordinates of the vehicle. In detail, the quaternion of the shooting device is (), where () is represented as q=w + xi + yj + zk. Substituting into Equation (2), the rotation matrix in Equation (1) can be obtained. Equation (2) is the coordinates of this region relative to the global coordinate system after the quaternion of the shooting device is transformed into Equation (1).

[0046] S190 establishes a three-dimensional map of farmland and agricultural products. If a plurality of images 170 containing farmland in a reserved area have corresponding feature coordinates, and another plurality of images 170 containing farmland in a reserved area also have corresponding feature coordinates, the plurality of images 170 containing farmland in a reserved area have common feature coordinates and are superimposed to establish a three-dimensional map of farmland and agricultural products, and display the agricultural products 120 in the reserved area 160. Specifically, since there may be some overlap in each farmland image captured by the imaging device 20, in the coordinate conversion step of S170, it is calculated that a plurality of images 170 containing farmland in a reserved area have corresponding feature coordinates, and another plurality of images 170 containing farmland in a reserved area also have corresponding feature coordinates, and their common feature coordinates are superimposed. In this way, the above description is repeated to construct a three-dimensional map of farmland and agricultural products.

[0047] The three-dimensional farmland and crop distribution map is based on the fact that after the uprooting action, at least one preserved area 160 is retained, and the distance that the vehicle 10 moves along the direction of the farmland corresponds to the preserved area 160. Specifically, as the vehicle 10 moves along the direction of the farmland, the imaging device 20 captures farmland images according to its moving position, thus obtaining multiple images 170 of farmland including the preserved area. By superimposing the common feature coordinates of the multiple images 170 of farmland including the preserved area, a three-dimensional farmland and crop distribution map can be established.

[0048] Furthermore, the three-dimensional farmland and crop distribution map displays the crops 120 in the preserved area 160 based on the distance the vehicle 10 moves along the direction of the farmland and after the uprooting action. Specifically, as the vehicle 10 moves along the direction of the farmland, the imaging device 20 captures farmland images of the corresponding area according to its moving position, thereby obtaining multiple images 170 of the farmland including the preserved area, and a three-dimensional farmland and crop distribution map is established through feature coordinates.

[0049] However, since the three-dimensional farmland crop distribution map is constructed through overlapping areas, in order to avoid repeatedly marking the crop 120 in the reserved area 160, the center distance D set in step S30 is used as the range for repetition judgment. The center distance between the reserved crops after the farmland image is identified by thinning should be greater than or equal to half of the center distance D. If the center distance between the overlapping areas is less than half of the center distance D, it is determined to be the same crop 120 in the reserved area 160. By analogy, the number of crops in the farmland after thinning and weeding can be calculated in order to manage the growth status of the crops 120. The three-dimensional farmland crop distribution map is projected to form a two-dimensional farmland crop distribution map.

[0050] Data is output from S170 to S190 to establish a three-dimensional farmland and crop distribution map. As shown in Figure 9A, multiple images 170 of farmland including the reserved area are obtained by moving the vehicle 10 along the direction of farmland. Through the above description, a three-dimensional coordinate farmland and crop distribution map can be constructed by using the common feature coordinates in the overlapping areas.

[0051] Figure 9B is a two-dimensional map of crop distribution in farmland constructed by projecting Figure 9A. To avoid duplicate marking of crop 120 in the reserved area 160, and since thinning and weeding have already been completed in the farmland, half the center distance D is set for duplicate determination. If the planting distance between crop 120 is less than half the center distance D, this area is considered as one crop, and this is used to estimate. In this way, when converted into a two-dimensional map of crop distribution in farmland, it is easier to understand the total estimated crop quantity in the farmland after thinning and weeding, so as to facilitate subsequent farmland management.

[0052] In addition, the pixel coordinates of crops 120 and weeds 130 in the farmland image need to be converted into three-dimensional coordinates so that the operating arm 50 of the vehicle 10 can perform the pulling action in the designated area of ​​the farmland to thin out seedlings and remove weeds, as shown in Figure 7. The method of converting different dimensions and coordinate positions will be further explained in the following content.

[0053] S200 obtains pixel coordinates. The recognition bounding box 80 of the center point recognition frame image 110 has a center point position 100, and the corresponding pixel coordinates are obtained. Specifically, since the shooting device 20 is a depth camera, its lens has an infrared light source. By calculating the distance between the lens and the object through the reflection time between infrared rays, the depth value is obtained. Therefore, the depth value and pixel coordinates of the center point of each weed 130 and each crop 120 can also be obtained in the captured farmland image.

[0054] S210 converts the three-dimensional spatial coordinates. Based on the pixel coordinates and depth values, the three-dimensional spatial coordinates are obtained by performing a three-dimensional coordinate formula conversion. The pixel coordinates of crop 120 and weed 130 and their respective depth values ​​are converted by the intrinsic parameters as shown in Equation (3). Equation (3) is , where is the pixel coordinate of the target point, where the target points are crop 120 and weed 130, and are the principal point coordinates of the lens, and and are the focal lengths of the X-axis and Y-axis, respectively, and d is the depth value. After substituting the pixel coordinates of crop 120 and weed 130 into Equation (3), the corresponding three-dimensional spatial coordinates (X,Y,Z) are obtained.

[0055] S220 converts the three-dimensional coordinates of the vehicle. The three-dimensional coordinates of the vehicle are obtained according to the matrix transformation formula of the three-dimensional spatial coordinates and the extrinsic parameters. The coordinate system of the shooting device 20 and the coordinate system of the vehicle 10 are in a geometric spatial relationship. The coordinate system transformation between them is established through the extrinsic parameters as shown in Equation (4). Equation (4) is: Substituting the three-dimensional spatial coordinates (X,Y,Z) into Equation (4), the coordinates (X',Y',Z') of the vehicle are obtained so that the operating arm 50 of the vehicle 10 can perform the pulling action in the designated area, removing the crops 120 and weeds 130 in the pulling area 150, realizing automated thinning and weeding, thus reducing labor costs and constructing a three-dimensional farmland crop distribution map through the coordinates of the crops 120. Where R is the rotation matrix, i represents the X, Y and Z axis coordinates after rotation, and j represents the original X, Y and Z axis coordinates; T is the translation vector, and , and are the displacements along the X, Y and Z axes, respectively.

[0056] In summary, the present invention provides a method and system for judging crops and weeds based on machine vision. It can identify crops 120 and weeds 130 in farmland images by using object detection model 30. At the same time, based on the growth range between crops 120, it can quickly perform thinning and weeding. After completing the thinning and weeding of the farmland, it can construct a three-dimensional distribution map of crops in the farmland. Through this map, it can better control the yield of crops 120 and effectively improve the efficiency of agricultural management, so as to achieve the purpose of the present invention. [Simplified Explanation of the Diagram]

[0057] Figure 1 is a block diagram of a system for judging crops and weeds based on machine vision according to the present invention. Figure 2 is a flowchart of a method for judging crops and weeds based on machine vision according to the present invention. Figure 3 is a partial flowchart of a method for judging crops and weeds based on machine vision according to the present invention. Figure 4 is a partial flowchart of a method for judging crops and weeds based on machine vision according to the present invention. Figure 5 is a partial flowchart of a method for judging crops and weeds based on machine vision according to the present invention. Figure 6 is a partial flowchart of a method for judging crops and weeds based on machine vision according to the present invention. Figures 7A and 7B are schematic diagrams of the present invention with identification bounding boxes. Figure 8 is a schematic diagram of the present invention with crops and weeds, wherein crops are divided into retention areas and removal areas. Figure 9A is a schematic diagram of a three-dimensional spatial distribution map of farmland crops according to the present invention. Figure 9B is a schematic diagram of a two-dimensional spatial distribution map of farmland crops constructed by projecting Figure 9B according to the present invention. [Biomaterial Storage]

[0059] None

Claims

1. A method for identifying crops and weeds based on machine vision, comprising the following steps: capturing an image, wherein a vehicle has a shooting device, the vehicle moves along the direction of a farmland, and the shooting device captures an image of the farmland; identifying crops, inputting the farmland image into an object detection model, and after identification by the object detection model, defining and classifying at least one object in the farmland image as at least one crop and at least one weed, to form an identification image containing the crops and weeds; determining a removal area and a retention area, performing a center point calculation based on the identification image containing the crops and weeds to obtain a farmland image with a center point, and defining at least one removal area and at least one retention area based on the farmland image with the center point, wherein the center point calculation further includes setting a set distance value, and setting a center distance between the crops; If the area is to be removed, and the distance between the weeds and crops in the farmland image with the center point is less than the center distance, the operating arm of the vehicle performs a removal action to remove the crops and weeds in the removal area; and if the area is to be retained, and the distance between the crops in the farmland image with the center point is greater than or equal to the center distance, the operating arm of the vehicle does not perform the removal action in the retained areas.

2. A method for judging crops and weeds based on machine vision as described in claim 1, further comprising constructing a three-dimensional spatial map of crop distribution in farmland, wherein the imaging device moves along the direction of the farmland according to the vehicle to obtain a plurality of images of farmland containing the reserved areas, and the plurality of images of farmland containing the reserved areas are subjected to a three-dimensional spatial transformation process to construct a three-dimensional spatial map of crop distribution in farmland.

3. A method for identifying crops and weeds based on machine vision as described in claim 1, wherein the step of crop identification further includes the following steps: identifying and classifying objects, using the object detection model to identify and classify the objects in the farmland image, and marking each object with an identification bounding box, and forming an identified object within the identification bounding box; contour segmentation, performing contour detection on the identified object to obtain an object contour; and center localization, obtaining a center point based on the centroid of the object contour to form an identification box image with a center point.

4. A method for identifying crops and weeds based on machine vision as described in claim 1, wherein the method for training the object detection model further includes the following steps: labeling and classifying images, labeling the objects in the farmland image, and defining the categories as crops and weeds respectively; augmenting the farmland image, performing image augmentation on the labeled farmland image to obtain multiple labeled farmland images; and outputting the object detection model, inputting the labeled farmland images into a machine learning model for training, and outputting the object detection model.

5. A machine vision-based method for identifying crops and weeds, as described in claim 1 or claim 3, further comprising the following steps: obtaining pixel coordinates, wherein the identification bounding box of the image of the center point has a center point position and a depth value, and obtaining the corresponding pixel coordinates; transforming the three-dimensional spatial coordinates, obtaining a three-dimensional spatial coordinate by performing a three-dimensional coordinate formula transformation based on the pixel coordinates and the depth value; and transforming the three-dimensional coordinates of the vehicle, obtaining the three-dimensional coordinates of the vehicle based on the three-dimensional spatial coordinates and the extrinsic parameter matrix transformation formula.

6. A method for determining crops and weeds based on machine vision as described in claim 2, wherein the three-dimensional farmland crop distribution map can be projected to form a two-dimensional farmland crop distribution map.

7. A method for judging crops and weeds based on machine vision as described in claim 2, wherein the three-dimensional spatial transformation process further includes the following steps: outputting data, wherein the output of multiple images of farmland containing the reserved areas has a timestamp, a world coordinate of a shooting device, a quaternion of a shooting device, and the multiple images of farmland containing the reserved areas are formed by the shooting device capturing different farmland images in different areas of the farmland when the vehicle moves on the farmland, and the multiple images of the farmland are aggregated from the different farmland images; The imaging device performs coordinate transformation based on the timestamp and the world coordinates of the imaging device to form a coordinate transformation matrix. It then outputs a feature coordinate for each of the preserved areas and the coordinate transformation matrix. Finally, it establishes a three-dimensional map of farmland and crop distribution. If a portion of the multiple images containing the farmland in the preserved areas has the corresponding feature coordinate, and another portion of the multiple images also has the corresponding feature coordinate, the multiple images containing the farmland in the preserved areas share the same feature coordinate and are superimposed to create a three-dimensional map of farmland and crop distribution, displaying the crops in the preserved areas.

8. A system for identifying crops and weeds based on machine vision, comprising: A vehicle that can move on farmland; A camera device, mounted on the vehicle, captures an image of the farmland based on the position of the vehicle moving over the farmland; an object detection model identifies at least one object in the farmland image and classifies and defines the objects as at least one crop and at least one weed; a removal judgment module sets a center distance based on the identified farmland image and the removal judgment module, determines whether it is a retention area or a removal area based on the center distance, transmits a coordinate command to an operating arm of the vehicle to perform a removal action, and removes the crops and weeds in the removal area; and a coordinate conversion module is used to convert the coordinates, converting the pixel coordinates of the farmland image into a three-dimensional coordinate of the camera device, and then converting the three-dimensional coordinates into the three-dimensional coordinates of the vehicle.

9. A machine vision-based system for identifying crops and weeds as described in claim 8, wherein the imaging device is a depth camera (RGB-D) for capturing a depth image of the farmland and an RGB image of the farmland.

10. A system for identifying crops and weeds based on machine vision as described in claim 8, wherein the object detection model is based on a machine learning recognition model.

11. A system for determining crops and weeds based on machine vision as described in claim 8, further comprising a map construction module, wherein the imaging device moves along the direction of the farmland according to the vehicle to obtain a plurality of images of the farmland containing the reserved areas, and the plurality of images of the farmland containing the reserved areas have the reserved areas, and a three-dimensional spatial map of the distribution of crops in the farmland is constructed through the reserved areas and a coordinate transformation matrix.

Images