Crop weed identification and positioning method and system

Abstract

The invention belongs to the technical field of crop weed recognition and positioning, and particularly provides a crop weed recognition and positioning method and system, and the method comprises the steps: collecting a data set containing a weed image; training is carried out under a YOLOv8 framework of ultralytics based on the data set, and a weight file capable of being used for weed recognition is generated; integrating the trained weight file into a YOLOv8 framework to generate a YOLOv8 detection model; acquiring an image captured by the real depth camera, and performing weed target detection on the image captured by the real depth camera by using the YOLOv8 detection model; extracting pixel coordinates of a central point of each target detection frame from a detection result; pixel coordinates extracted in the camera coordinate system are converted into space actual coordinates in the lower computer coordinate system. Various defects of a traditional manual weeding mode can be overcome, and the agricultural production efficiency is improved.

Description

Background Art

[0002] In modern agricultural production, weed management is not only related to the healthy growth of crops, but also the cornerstone of improving overall agricultural output and optimizing resource utilization. However, facing the vast farmland and complex and changing growth environment, traditional weed control methods not only consume a lot of manpower and material resources, but are also inefficient, and the cost burden is even more unbearable when facing large-scale agricultural production.

[0003] The rapid development of science and technology, especially the breakthroughs in computer vision and deep learning technology, has provided a new perspective for solving this problem. Image recognition technology, as a core component of computer vision, can simulate human visual perception and automatically identify and locate targets in images. The introduction of this technology has made it possible to automatically identify and locate crops and weeds, bringing revolutionary changes to agricultural production.

[0004] However, existing image recognition methods often face many challenges in practice. On the one hand, insufficient recognition accuracy may lead to misjudgment and missed judgment, affecting the weed control effect; on the other hand, the problem of poor real-time performance makes it difficult to meet the needs of rapid response in agricultural production. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a crop weed identification and positioning method and system.

[0006] In a first aspect, the technical solution of the present invention provides a method for identifying and locating crop weeds, comprising the following steps: Collect a dataset containing weed images; Based on the dataset, training was performed under the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification; Integrate the trained weight file into the YOLOv8 framework to generate a YOLOv8 detection model; Get the image captured by the realsense depth camera and use the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; Extract the pixel coordinates of the center point of each target detection box from the detection results; The pixel coordinates extracted in the camera coordinate system are converted into actual spatial coordinates in the lower computer coordinate system.

[0007] As a further limitation of the technical solution of the present invention, the step of collecting a data set containing weed images includes: Collect a dataset containing weed images; Set one or more labels for each image in the dataset to identify the type and location of the weeds in the image; Use the annotation tool to annotate the weeds in the image and convert the annotations to YOLO format.

[0008] As a further limitation of the technical solution of the present invention, the steps of training the data set under the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification include: Modify the YOLOv8 configuration file; Start training in the YOLOv8 framework of ultralytics by loading the configuration file of the dataset, the YOLOv8 configuration file, and the pre-trained weights; After the training is completed, a weight file that can be used for weed identification is generated in the specified output directory.

[0009] As a further limitation of the technical solution of the present invention, before the step of integrating the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model, the following steps are included: Determine whether the YOLOv8 framework and required dependent libraries are installed and configured on the local computer or target device; If yes, execute the steps: Integrate the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; If not, install and configure the YOLOv8 framework and required dependencies on your local computer or target device.

[0010] As a further limitation of the technical solution of the present invention, the steps of acquiring an image captured by a realsense depth camera and performing target detection of weeds on the image captured by the realsense depth camera using a YOLOv8 detection model include: Initialize the YOLOv8 detection model using the weight file; Get the image captured by the realsense depth camera and get a pair of coherent frames, including a depth frame and a color frame; Convert the depth frame and color frame data into NumPy arrays; Use the initialized YOLOv8 detection model to perform target detection on the color image and draw the detection results on the original color image; the detection results contain a list of multiple detection boxes, each of which contains the category, confidence, and bounding box coordinates; Convert the depth image to a color image; The processed color image and the color map of the depth image are stacked horizontally for display.

[0011] As a further limitation of the technical solution of the present invention, the step of extracting the pixel coordinates of the center point of each target detection frame from the detection result includes: Traverse the detection results and calculate the coordinates of the center point of each detection box; Draw the detection frame and center point on the original color image, and output the coordinate information to complete the identification and positioning of the weeds.

[0012] As a further limitation of the technical solution of the present invention, the step of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system includes: The pixel coordinates extracted from the camera coordinate system are converted to the image physical coordinate system based on the physical size of the pixel and the coordinates of the image center; Convert the image physical coordinate system to the camera coordinate system; Based on the calibration parameters of the realsense depth camera, the coordinates are converted from the camera coordinate system to the world coordinate system to obtain the actual spatial coordinates in the lower computer coordinate system.

[0013] As a further limitation of the technical solution of the present invention, after the step of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system, the following step is further included: The actual spatial coordinates acquired are transmitted to the three-degree-of-freedom gimbal control system through the serial port to complete the benchmarking process.

[0014] In a second aspect, the technical solution of the present invention also provides a crop weed identification and positioning system, including a data collection module, a weight file training module, a detection model generation module, a positioning detection module, a coordinate acquisition module, and a coordinate conversion module; A data collection module collects a dataset containing weed images; The weight file training module is trained based on the dataset in the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification; The detection model generation module integrates the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; The positioning detection module obtains the image captured by the realsense depth camera and uses the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; The coordinate acquisition module extracts the pixel coordinates of the center point of each target detection box from the detection results; The coordinate conversion module converts the pixel coordinates extracted from the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system.

[0015] As a further limitation of the technical solution of the present invention, the system also includes a coordinate output module, which transmits the acquired actual spatial coordinates to the three-degree-of-freedom gimbal control system through the serial port to complete the calibration process.

[0016] As a further limitation of the technical solution of the present invention, the data collection module is specifically used to collect a data set containing weed images; set one or more labels for each image in the data set to identify the type and location of the weeds in the image; use a labeling tool to label the weeds in the image, and convert the labeling into a YOLO format.

[0017] As a further limitation of the technical solution of the present invention, the weight file training module is specifically used to modify the YOLOv8 configuration file; start training by loading the configuration file of the data set, the YOLOv8 configuration file and the pre-trained weights under the YOLOv8 framework of ultralytics; after the training is completed, a weight file that can be used for weed identification is generated in a specified output directory.

[0018] As a further limitation of the technical solution of the present invention, the system also includes an environment detection module for determining whether the YOLOv8 framework and the required dependent libraries are installed and configured on the local computer or target device; if so, triggering the weight file training module; if not, installing and configuring the YOLOv8 framework and the required dependent libraries on the local computer or target device.

[0019] As a further limitation of the technical solution of the present invention, the positioning detection module is specifically used to initialize the YOLOv8 detection model using a weight file; obtain an image captured by a realsense depth camera, and obtain a pair of coherent frames, including a depth frame and a color frame; convert the data of the depth frame and the color frame into a NumPy array; use the initialized YOLOv8 detection model to perform target detection on the color image and draw the detection result on the original color image; the detection result includes a list of multiple detection boxes, each detection box includes a category, a confidence level, and a bounding box coordinate; convert the depth image into a color map; and horizontally stack the processed color image and the color map of the depth image for display.

[0020] As a further limitation of the technical solution of the present invention, the coordinate acquisition module traverses the detection results and calculates the coordinates of the center point of each detection frame; draws the detection frame and the center point on the original color image, and outputs the coordinate information to complete the identification and positioning of the weeds.

[0021] As a further limitation of the technical solution of the present invention, the coordinate conversion module is specifically used to convert the pixel coordinates extracted in the camera coordinate system into the physical coordinate system of the image based on the physical size of the pixel and the coordinates of the image center; convert the physical coordinate system of the image into the camera coordinate system; based on the calibration parameters of the realsense depth camera, convert the coordinates in the camera coordinate system to the world coordinate system to obtain the actual spatial coordinates in the coordinate system of the lower computer.

[0022] It can be seen from the above technical solutions that the present invention has the following advantages: this method can not only overcome the various drawbacks of traditional manual weeding methods and improve agricultural production efficiency and yield, but also provide strong technical support for the development of precision agriculture and promote the development of modern agriculture in a more intelligent and refined direction. By collecting high-quality image data sets containing crops and weeds and training them under the YOLOv8 framework, the generated weight file can accurately identify weeds and crops in the image, greatly improving the recognition accuracy.

[0023] By leveraging the efficiency of the YOLOv8 framework and combining it with the real-time image capture capability of the realsense depth camera, this method can achieve real-time detection of crops and weeds, making it possible to remove weeds in a timely manner in agricultural production.

[0024] By training the model to learn image data under various environmental conditions, this method can better adapt to the influence of environmental factors such as different lighting and soil moisture, and improve the stability and reliability of recognition.

[0025] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0028] Figure 2 is a schematic block diagram of an apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the Ultralytics framework is an open source computer vision and deep learning framework designed to simplify the process of training, evaluating and deploying visual models. The framework is based on the PyTorch deep learning framework, has excellent performance and flexibility, and provides a series of optimization and acceleration functions to speed up training and reasoning. Ultralytics provides concise APIs and command-line tools that allow users to easily train, evaluate and deploy visual models.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a crop weed identification and positioning method, comprising the following steps: Step 1: Collect a dataset containing weed images; Step 2: Train the dataset using the YOLOv8 framework of Ultralytics to generate a weight file that can be used for weed identification. Step 3: Integrate the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; Step 4: Get the image captured by the realsense depth camera and use the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; Step 5: Extract the pixel coordinates of the center point of each target detection box from the detection results; Step 6: Convert the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system.

[0032] In some embodiments, the step of collecting a dataset comprising weed images includes: Step 11: Collect a dataset containing weed images. Step 12: Set one or more labels for each image in the dataset to identify the type and location of the weeds in the image; Step 13: Use the annotation tool to annotate the weeds in the image and convert the annotations to YOLO format.

[0033] In some embodiments, the step of training a weight file that can be used for weed identification based on a dataset under the YOLOv8 framework of ultralytics includes: Step 21: Modify the YOLOv8 configuration file; Step 22: Start training in the YOLOv8 framework of ultralytics by loading the configuration file of the dataset, the YOLOv8 configuration file, and the pre-trained weights; Step 23: After the training is completed, a weight file that can be used for weed identification is generated in the specified output directory.

[0034] It should be noted that the model files used for YOLO-v8 pre-training should be installed on the premise that the python interpreter is fully installed. After downloading the training model and deploying the training environment, create a new folder datas in the YOLO-v8-main\ultralytics\datasets folder to store the sample images of the data set and the corresponding labeled data, and create a new file mydata.yaml, configure the path for storing the data set, and set the training object and its code in the order of setting the label in the previous labeling. Next, put the sample image and labeled data text in the datas folder according to train and val respectively, and then configure the train.py file in the root directory, set the storage location of the official model and the corresponding parameters such as the number of training (epochs).

[0035] If Python is installed, select the Python interpreter and run train.py to train the model. If an error occurs during training and the file path is garbled, move the root directory of the training environment to a path without Chinese characters. After the program automatically exits after training the model, the real-time detection data and two models during the model training process are obtained. Select the best.pt model file in the weights folder as the weight file for this training.

[0036] It should be noted that before the step of integrating the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model, the step includes: determining whether the YOLOv8 framework and the required dependent libraries are installed and configured on the local computer or target device; if so, executing the step: integrating the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; if not, installing and configuring the YOLOv8 framework and the required dependent libraries on the local computer or target device.

[0037] In some embodiments, the steps of acquiring an image captured by a realsense depth camera and performing weed target detection on the image captured by the realsense depth camera using a YOLOv8 detection model include: Step 41: Initialize the YOLOv8 detection model using the weight file; Step 42: Get the image captured by the realsense depth camera and get a pair of consecutive frames, including a depth frame and a color frame; use pipeline.wait_for_frames() to wait and get a pair of consecutive frames; check whether the depth frame and color frame are successfully obtained. If not, skip the rest of the current loop and continue to the next loop.

[0038] Step 43: Convert the data of the depth frame and the color frame into NumPy arrays; Step 44: Use the initialized YOLOv8 detection model to perform target detection on the color image and draw the detection results on the original color image; the detection results include a list of multiple detection boxes, each detection box includes a category, a confidence level, and a bounding box coordinate; In this step, check whether the detection results (results) contain a render method. If so, call this method to draw the detection results on the original color image.

[0039] Step 45: Convert the depth image to a color map; use OpenCV’s applyColorMap function and the COLORMAP_JET color map to better visualize the depth information.

[0040] Step 46: The processed color image and the color map of the depth image are stacked horizontally for display.

[0041] In some embodiments, the step of extracting the pixel coordinates of the center point of each target detection box from the detection result includes: Step 51: Traverse the detection results and calculate the coordinates of the center point of each detection frame; Step 52: Draw a detection frame and a center point on the original color image, and output the coordinate information to complete the recognition and positioning of the weeds.

[0042] It should be noted that the pixel coordinate system is a two-dimensional image coordinate system constructed in units of pixels, with the upper left corner of the image as the origin, the horizontal axis as the u-axis, the vertical axis as the v-axis.

[0043] Image physical coordinate system: A two-dimensional image coordinate system with the camera optical center as the origin and m as the unit (usually millimeters), which is used to describe the physical position of a point on the image.

[0044] Camera coordinate system: With the camera optical center as the origin, the camera optical axis as the Z axis, and the X and Y axes parallel to the X and Y axes of the image physical coordinate system, it is used to describe the three-dimensional position of objects in the camera field of view.

[0045] World coordinate system (lower computer coordinate system): a global, fixed reference coordinate system used to describe the three-dimensional position of objects in the real world.

[0046] In some embodiments, the step of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system includes: Step 61: based on the physical size of the pixel and the coordinates of the center of the image, the pixel coordinates extracted in the camera coordinate system are converted into the image physical coordinate system; This conversion usually involves the physical size of the pixel ( ) and the image center ( ) coordinates. The conversion formula is as follows:

[0047] Among them, (u, v) is the pixel coordinate and (x, y) is the image physical coordinate.

[0048] Step 62: Convert the image physical coordinate system into the camera coordinate system; This conversion is based on the imaging principle of the camera, namely the pinhole imaging model. The conversion involves the focal length f of the camera and the principle of similar triangles. However, in this step, since the image physical coordinate system and the camera coordinate system share the X-axis and Y-axis, and the origin is at the optical center of the camera, this step is mainly to add the Z axis (depth information) to the coordinates. This usually requires obtaining the Z value through other methods (such as stereo vision, depth sensor, etc.).

[0049] In the case of only a monocular camera and no additional depth information, it is impossible to directly obtain the Z value in the camera coordinate system from the physical coordinates of the image. However, in some applications, the Z value can be estimated through assumptions (such as the plane assumption) or prior knowledge.

[0050] Step 63: Based on the calibration parameters of the realsense depth camera, the coordinates are converted from the camera coordinate system to the world coordinate system to obtain the actual spatial coordinates in the lower computer coordinate system; This transformation involves the camera's external parameter matrix, including the rotation matrix R and the translation vector T. The transformation formula is as follows:

[0051] in,( ) is the coordinate in the camera coordinate system, ( ) are the coordinates in the world coordinate system.

[0052] It should be noted that the above conversion process requires the camera's intrinsic parameter matrix (focal length f, pixel size , Image Center ) and external parameter matrices (rotation matrix R, translation vector T). These parameters are usually obtained through the camera calibration process.

[0053] In a monocular camera system, obtaining accurate depth information is a challenge. Without additional sensors or prior knowledge, it is difficult to recover the complete 3D information directly from the image. When performing coordinate conversion, it is necessary to ensure that the definitions and units of each coordinate system are consistent to avoid errors. In some embodiments, after the step of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system, the step further includes: Step 7: Transmit the acquired actual spatial coordinates to the three-DOF gimbal control system through the serial port to complete the calibration process.

[0054] like Figure 2 As shown, an embodiment of the present invention also provides a crop weed identification and positioning system, including a data collection module, a weight file training module, a detection model generation module, a positioning detection module, a coordinate acquisition module, and a coordinate conversion module; A data collection module collects a dataset containing weed images; The weight file training module is trained based on the dataset in the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification; The detection model generation module integrates the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; The positioning detection module obtains the image captured by the realsense depth camera and uses the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; The coordinate acquisition module extracts the pixel coordinates of the center point of each target detection box from the detection results; The coordinate conversion module converts the pixel coordinates extracted from the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system.

[0055] The system also includes a coordinate output module, which transmits the acquired actual spatial coordinates to the three-degree-of-freedom gimbal control system through the serial port to complete the calibration process.

[0056] In some embodiments, the data collection module is specifically used to collect a data set containing weed images; set one or more labels for each image in the data set to identify the type and location of the weeds in the image; use a labeling tool to label the weeds in the image, and convert the labeling into a YOLO format.

[0057] In some embodiments, the weight file training module is specifically used to modify the YOLOv8 configuration file; start training by loading the configuration file of the data set, the YOLOv8 configuration file and the pre-trained weights under the YOLOv8 framework of ultralytics; after the training is completed, generate a weight file that can be used for weed identification in a specified output directory.

[0058] In some embodiments, the system also includes an environment detection module for determining whether the YOLOv8 framework and required dependent libraries are installed and configured on the local computer or target device; if so, triggering the weight file training module; if not, installing and configuring the YOLOv8 framework and required dependent libraries on the local computer or target device.

[0059] In some embodiments, the positioning detection module is specifically used to initialize the YOLOv8 detection model using a weight file; obtain an image captured by a realsense depth camera, and obtain a pair of coherent frames, including a depth frame and a color frame; convert the data of the depth frame and the color frame into a NumPy array; use the initialized YOLOv8 detection model to perform target detection on the color image and draw the detection result on the original color image; the detection result includes a list of multiple detection boxes, each detection box includes a category, a confidence level, and a bounding box coordinate; convert the depth image into a color map; and horizontally stack the processed color image and the color map of the depth image for display.

[0060] In some embodiments, the coordinate acquisition module traverses the detection results, calculates the coordinates of the center point of each detection frame, draws the detection frame and the center point on the original color image, and outputs the coordinate information to complete the identification and positioning of the weeds.

[0061] In some embodiments, the coordinate conversion module is specifically used to convert the pixel coordinates extracted in the camera coordinate system into the physical coordinate system of the image based on the physical size of the pixel and the coordinates of the center of the image; convert the physical coordinate system of the image into the camera coordinate system; based on the calibration parameters of the realsense depth camera, convert the coordinates in the camera coordinate system to the world coordinate system to obtain the actual spatial coordinates in the coordinate system of the lower computer.

[0062] In the specific application process, a dot is drawn in the image captured by the depth camera based on the fixed point automatically given by YOLOv8, and the dot is located at the center of the rectangle determined by the given fixed point; a variable named Center is created to represent the center coordinates of the two given points and the cv2.putText function is used to mark them on the image, and the marked center coordinates are displayed; the coordinate information of the four boundary points and the center position of the detection frame is printed; the trained weight file is used to detect the weed crop pictures in various postures, and the detection results and the coordinates of the center point of the detection frame are output, and the coordinate information is output in the PyCharm terminal to complete the recognition and positioning of the weeds and obtain the coordinate data.

[0063] Get the YOLOv8 package source code from Github, create a virtual environment named pythonProject3 from Pycharm, and install other required modules such as pytorch, CUDA, and CUDNN using the cmd command prompt and pycharm terminal.

[0064] The crop in the embodiment of the present invention can be pepper. The Jetson Nano B01 development board is used as the hardware operating environment for real-time linkage between ultralytics and realsense depth cameras. Through the analysis of the overall YOLO framework, the part that controls the output label frame of the detection frame is found from the util directory, and the following code structure is added to the plotting module: First, a red dot is drawn in the predicted image (the image captured by the depth camera). The dot is located at the center position of the rectangle determined by two (fixed points automatically given by YOLOv8, set as the default setting of the recognition system, and the rectangle is obtained by YOLOv8 plus functional code and weight file.) The radius of the circle is determined by the variable text. Secondly, a variable named Center is created to represent the center coordinates of the two given points ((p1, p2)) and the cv2.putText function is used to mark them on the image and display them near the center dot. Finally, the coordinate information of the four boundary points and the center point is printed. Then, the trained weight file is used to detect the weed pepper in various postures, and the detection results and the coordinates of the center point of the detection frame are successfully output in the image (the rectangle is obtained by YOLOv8 plus functional code and weight file), and the coordinate information is output in the PyCharm terminal to complete the identification and positioning of the weeds and obtain the coordinate data. (First, place the trained weight file in the .venv root directory, and then obtain the absolute path of the file as the model function in the model calling code. Then put the video stream and image stream source to be detected in the specified location and obtain its absolute path as the source function. Among them, "model" is the custom pepper-weed model weight file called, "source" is the target detection object used in the model prediction, and "project" is the storage location of the detection results after the project is run. Subsequently, the pictures to be detected are placed in the assets folder, and all images in the model file that appear in the image can be effectively displayed and the detection frame can be successfully rendered).

[0065] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0066] The embodiments of the crop weed identification and positioning system disclosed in the embodiments of the present invention belong to the same inventive concept as the crop weed identification and positioning methods of the above-mentioned embodiments. For details not fully described in the embodiments of the crop weed identification and positioning system, reference can be made to the embodiments of the above-mentioned crop weed identification and positioning methods.

[0067] The crop weed identification and positioning system is a unit and algorithm step of each example described in combination with the embodiments disclosed herein, and can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0068] Those skilled in the art will appreciate that various aspects of the crop weed identification and positioning method can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which can be collectively referred to as "circuit", "module" or "system".

[0069] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A method for identifying and locating crop weeds, It is characterized in that The steps include: Collect a dataset containing weed images; Based on the dataset, training was performed under the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification; Integrate the trained weight file into the YOLOv8 framework to generate a YOLOv8 detection model; Get the image captured by the realsense depth camera and use the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; Extract the pixel coordinates of the center point of each target detection box from the detection results; The pixel coordinates extracted in the camera coordinate system are converted into actual spatial coordinates in the lower computer coordinate system.

2. The crop weed identification and positioning method according to claim 1, It is characterized in that The steps to collect a dataset containing weed images include: Collect a dataset containing weed images; Set one or more labels for each image in the dataset to identify the type and location of the weeds in the image; Use the annotation tool to annotate the weeds in the image and convert the annotations to YOLO format.

3. The crop weed identification and positioning method according to claim 2, It is characterized in that Based on the dataset, the training is performed under the YOLOv8 framework of ultralytics. The steps to generate a weight file that can be used for weed identification include: Modify the YOLOv8 configuration file; Start training in the YOLOv8 framework of ultralytics by loading the configuration file of the dataset, the YOLOv8 configuration file, and the pre-trained weights; After the training is completed, a weight file that can be used for weed identification is generated in the specified output directory.

4. The crop weed identification and positioning method according to claim 3, It is characterized in that Integrate the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model. The steps include: Determine whether the YOLOv8 framework and required dependent libraries are installed and configured on the local computer or target device; If yes, execute the steps: Integrate the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; If not, install and configure the YOLOv8 framework and required dependencies on your local computer or target device.

5. The crop weed identification and positioning method according to claim 4, It is characterized in that The steps of obtaining an image captured by a realsense depth camera and using a YOLOv8 detection model to perform weed object detection on the image captured by the realsense depth camera include: Initialize the YOLOv8 detection model using the weight file; Get the image captured by the realsense depth camera and get a pair of coherent frames, including a depth frame and a color frame; Convert the depth frame and color frame data into NumPy arrays; Use the initialized YOLOv8 detection model to perform target detection on the color image and draw the detection results on the original color image; the detection results contain a list of multiple detection boxes, each of which contains the category, confidence, and bounding box coordinates; Convert the depth image to a color image; The processed color image and the color map of the depth image are stacked horizontally for display.

6. The crop weed identification and positioning method according to claim 5, It is characterized in that The steps of extracting the pixel coordinates of the center point of each target detection box from the detection results include: Traverse the detection results and calculate the coordinates of the center point of each detection box; Draw the detection frame and center point on the original color image, and output the coordinate information to complete the identification and positioning of the weeds.

7. The crop weed identification and positioning method according to claim 6, It is characterized in that The steps of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system include: The pixel coordinates extracted from the camera coordinate system are converted to the image physical coordinate system based on the physical size of the pixel and the coordinates of the image center; Convert the image physical coordinate system to the camera coordinate system; Based on the calibration parameters of the realsense depth camera, the coordinates are converted from the camera coordinate system to the world coordinate system to obtain the actual spatial coordinates in the lower computer coordinate system.

8. The crop weed identification and positioning method according to claim 7, It is characterized in that After the step of converting the pixel coordinates extracted in the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system, the following steps are also included: The actual spatial coordinates acquired are transmitted to the three-degree-of-freedom gimbal control system through the serial port to complete the benchmarking process.

9. A crop weed identification and positioning system, It is characterized in that It includes data collection module, weight file training module, detection model generation module, positioning detection module, coordinate acquisition module and coordinate conversion module; A dataset collection module collects datasets containing weed images; The weight file training module is trained based on the dataset in the YOLOv8 framework of ultralytics to generate a weight file that can be used for weed identification; The detection model generation module integrates the trained weight file into the YOLOv8 framework to generate the YOLOv8 detection model; The positioning detection module obtains the image captured by the realsense depth camera and uses the YOLOv8 detection model to perform weed target detection on the image captured by the realsense depth camera; The coordinate acquisition module extracts the pixel coordinates of the center point of each target detection box from the detection results; The coordinate conversion module converts the pixel coordinates extracted from the camera coordinate system into the actual spatial coordinates in the lower computer coordinate system.

10. The crop weed identification and positioning system according to claim 9, It is characterized in that The system also includes a coordinate output module, which transmits the acquired actual spatial coordinates to the three-degree-of-freedom gimbal control system through the serial port to complete the calibration process.

Images