A crop subject identification and positioning method based on machine vision and application thereof

Abstract

The application discloses a crop subject recognition and positioning method based on machine vision and application, and relates to the technical field of recognition and positioning. An image set of a target region is acquired, feature extraction is performed on the image set to obtain feature labels, and the feature labels are added to the image set to obtain a target image set; semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data, a three-dimensional target region is obtained by modeling the subject according to the spatial point cloud data; a final pose observation matrix is generated according to the three-dimensional target region, and the camera position is updated according to the final pose observation matrix. The feature extraction and semantic segmentation accurately extract the information of tomatoes in the target image, the three-dimensional target region is obtained by modeling, and then the final pose observation matrix is generated to update the camera position. The problems that the flower stem is easily shielded and the immature tomatoes are mispicked are solved, the detection precision and accuracy of the picking point are improved, and the tomatoes stem is prevented from being damaged in the impurity removing process.

Description

Technical Field

[0001] This invention relates to the field of identification and positioning technology, specifically to a method and application for identifying and locating crop subjects based on machine vision. Background Technology

[0002] In the field of agricultural automation, the development of tomato harvesting technology has attracted much attention. Traditional harvesting methods rely on manual labor, which is inefficient and labor-intensive. In recent years, with the advancement of robotics technology, automated harvesting has gradually become a research hotspot. However, the slender characteristics of tomato flower stalks and inconsistent ripening levels pose many challenges to robotic harvesting.

[0003] Patent No. CN119006594A discloses a fruit picking recognition and localization method based on an ORB-SLAM model. A monocular camera is calibrated and corrected before being installed on a fruit picking robot. Images captured by the monocular camera during movement are input into an ORB-SLAM3 model, which outputs a visual map to determine the camera's position. Images of the fruit to be picked are captured by the monocular camera and input into a YOLOv8 model for identification and localization. The relative distance between the fruit picking robot and the fruit is determined for picking, ultimately achieving fruit picking recognition and localization.

[0004] Patent No.: CN118115879A. This invention involves collecting a large number of images of crops and associated weeds at different growth stages, establishing training, validation, and test sets, and training a first weed identification and localization model and a second weed identification and localization model. Real-time images of the area to be weeded are collected and input into the trained first and second weed identification and localization models respectively to obtain corresponding first and second weed identification and localization results. The first and second weed identification and localization results are combined to obtain the final weed identification and localization result, which is then used to drive a laser weeding device to burn the weeds.

[0005] In the existing technology, 1. Weeds other than tomato plants are removed by laser. Since the tomato plants are not identified, the stem area of ​​the tomato plants will be accidentally damaged during the laser weed removal process.

[0006] 2. Because tomato flower stalks are long and thin and easily obstructed by surrounding fruits, stems and other obstacles, the robot's detection accuracy at the picking point is greatly reduced. In addition, because tomatoes ripen at different times, unripe tomatoes are easily picked up incorrectly. Summary of the Invention

[0007] The purpose of this invention is to address the following issues mentioned in the background: 1. When using lasers to remove weeds other than tomato plants, the stems of the tomato plants may be accidentally damaged during the laser removal process because the tomato plants are not identified; 2. Due to the slender nature of tomato flower stalks, which are easily obscured by surrounding fruits, stems, and other obstacles, the detection accuracy of the robot at the picking point is greatly reduced, and unripe tomatoes are easily picked incorrectly because tomatoes ripen at different times. Therefore, this invention proposes a method and application for identifying and locating crop subjects based on machine vision.

[0008] A first aspect of this invention provides a method for identifying and locating crop subjects based on machine vision, the method comprising:

[0009] A set of images of the target region is obtained, and feature labels are extracted from the image set to obtain feature labels. The feature labels are then added to the image set to obtain a target image set. Each image in the image set contains a crop as the main subject. The crop as the main subject is a tomato plant. The feature labels include: a tomato plant label and a non-tomato plant label. The target image set contains multiple target images.

[0010] Semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data. The subject is then modeled based on the spatial point cloud data to obtain a three-dimensional target region. The target image set contains multiple target images.

[0011] A final pose observation matrix is ​​generated based on the three-dimensional target region, and the camera position is updated based on the final pose observation matrix.

[0012] Optionally, feature extraction is performed on the image set to obtain feature labels, including:

[0013] Feature labels are obtained by extracting features from the image set using a target preset model;

[0014] The target preset model includes a backbone network and a neck structure; the improvements of the target preset model compared to the YOLOv8 model include:

[0015] Replace the Conv modules in layers 1, 3, 5 and 7 of the backbone network with the target Conv modules, and replace the C2f modules in layers 2, 4 and 6 and 8 of the backbone network with the target C2f modules;

[0016] Replace each C2f module in the neck structure with a target C2f module, replace each Conv module in the neck structure with a target Conv module, add a target Conv module between the 12th and 13th layers, add a target Conv module between the 4th and 14th layers, add a target Conv module between the 6th and 11th layers, and add a target Conv module between the 9th and 10th layers;

[0017] A first attention module is added between layers 15 and 16 in the neck structure, a second attention module is added between layers 18 and 19 in the neck structure, and a third attention module is added after layer 21 in the neck structure to obtain the target preset model.

[0018] Optionally, the working principle of the target Conv module includes:

[0019] Obtain the first original feature map, determine the number of channels in the original feature map, and perform channel separation on the original feature map according to preset rules to obtain the first branch map and the second branch map;

[0020] The first branch graph is input into the Conv module to obtain the third branch graph, and the second branch graph is input into the DWConv module to obtain the fourth branch graph;

[0021] The third branch map and the fourth branch map are spliced ​​together to obtain an intermediate feature map, and a channel recombination operation is performed on the intermediate feature map to obtain the final feature map.

[0022] Optional, the working principle of the target C2f module includes:

[0023] Obtain a second original feature map, input the second original feature map into the Conv module to obtain a first feature map, input the first feature map into the PConv module to obtain a second feature map, input the second feature map into the PConv module to obtain a third feature map, concatenate the first feature map, the second feature map and the third feature map to obtain a fourth feature map, and input the fourth feature map into the Conv module to obtain a fifth feature map.

[0024] Optionally, semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data, including:

[0025] A classification mask is obtained by extracting target images from the target image set using a lightweight semantic segmentation network, and point cloud spatial point cloud data is generated based on the classification mask.

[0026] The formula for point cloud computing is:

[0027] ;

[0028] Among them, (X) i ,Y i Z i Let (u) be the coordinates of the i-th point in three-dimensional space. i ,v i Let D be the pixel coordinates of pixel i. i is the pixel depth value, s is a constant scaling factor, (c x ,c y ,f x ,f y ) represents the camera's intrinsic parameters, M(u) i ,v i ) is a pixel (u i ,v i The mask pixels are 1 for the main subject and 2 for the non-main subject.

[0029] Optionally, a three-dimensional target region is obtained by modeling the subject based on the spatial point cloud data, including:

[0030] Based on the spatial point cloud data, the main point cloud data is determined, and the main point cloud data is clustered to obtain a set of main clusters. The three-dimensional modeling data is calculated for each main cluster in the set of main clusters, and the three-dimensional target region is obtained by modeling based on the three-dimensional modeling data. The set of main clusters contains multiple main clusters. The three-dimensional modeling data includes center coordinates and radius.

[0031] Optionally, the final pose observation matrix is ​​generated based on the three-dimensional target region, including:

[0032] The equations of non-subject lines are determined based on the three-dimensional target region. The starting point coordinates and non-subject direction vectors are determined based on the equations of non-subject lines. The target center coordinates of the target subject cluster are obtained. The line parameters are calculated based on the starting point coordinates, the target center coordinates, and the non-subject direction vectors. The target subject cluster is any one of the subject cluster sets.

[0033] The perpendicular point coordinates are calculated based on the line parameters, the starting point coordinates, and the non-subject direction vector. The perpendicular point coordinates and the target center coordinates are then projected onto the world coordinate system to obtain the initial direction vector.

[0034] The initial direction vector is used as the Z-axis direction vector to determine the upward direction vector, and the upward direction vector is used as the Y-axis direction vector. The X-axis direction vector is obtained by performing a cross product of the Z-axis direction vector and the Y-axis direction vector. The initial pose matrix is ​​obtained by combining the X-axis direction vector, the Z-axis direction vector, the Y-axis direction vector, and the target center coordinates. The initial pose matrix is ​​then transformed according to preset settings to obtain the final pose observation matrix.

[0035] Optionally, the identification and positioning method is used to remove weeds from non-tomato plants, including weeds;

[0036] The specific process of removing impurities is as follows: the tomato plants are identified and located using an identification and positioning method;

[0037] The non-tomato plants are removed using a laser weeding device based on the labels on the non-tomato plants.

[0038] The beneficial effects of this invention are:

[0039] This invention proposes a machine vision-based method for identifying and locating crop subjects. It involves acquiring an image set of the target region, extracting features from the image set to obtain feature labels, and adding these labels to the image set to obtain a target image set. Semantic segmentation is then performed on each target image in the target image set to obtain spatial point cloud data. Based on the spatial point cloud data, a 3D target region is modeled. A final pose observation matrix is ​​generated based on the 3D target region, and the camera position is updated according to this matrix. Feature extraction and semantic segmentation accurately extract tomato information from the target image, modeling the 3D target region, and then generating the final pose observation matrix to update the camera position. This method solves the problems of thin, easily obscured flower stalks and accidental harvesting of unripe tomatoes, improves the accuracy and precision of harvesting point detection, ensures harvesting quality, and avoids accidental damage to tomato stems during weeding. Attached Figure Description

[0040] Figure 1 A flowchart of a method for identifying and locating crop subjects based on machine vision is provided for an embodiment of the present invention;

[0041] Figure 2 This invention provides a schematic diagram of the model structure of a machine vision-based crop subject identification and localization method. Detailed Implementation

[0042] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0043] This invention provides a method for identifying and locating crop subjects based on machine vision. See also... Figure 1 , Figure 1 A flowchart illustrating a machine vision-based method for identifying and locating crop subjects, provided in an embodiment of the present invention. The method includes the following steps:

[0044] S101, Obtain the image set of the target region, extract features from the image set to obtain feature labels, and add the feature labels to the image set to obtain the target image set;

[0045] S102, perform semantic segmentation on each target image in the target image set to obtain spatial point cloud data, and model the subject based on the spatial point cloud data to obtain the three-dimensional target region;

[0046] S103, the final pose observation matrix is ​​generated based on the three-dimensional target region, and the camera position is updated based on the final pose observation matrix.

[0047] Each image in the image set contains a crop as the main subject; the crop as the main subject is tomato plants; the feature labels include: tomato plant labels and non-tomato plant labels; the target image set contains multiple target images;

[0048] This invention provides a machine vision-based method for identifying and locating crop subjects. By employing feature extraction and semantic segmentation, it accurately extracts tomato information from target images, models a three-dimensional target region, and generates a final pose observation matrix to update the camera position. This method solves the problems of slender flower stalks being easily obscured and unripe tomatoes being mistakenly picked, improving the accuracy and precision of picking point detection and ensuring harvesting quality.

[0049] In one implementation, after acquiring an image set of the target region (i.e., images containing tomatoes and stems), feature extraction is performed on the image set to obtain feature labels (tomato plant labels, i.e., classification masks). These feature labels are then added to the image set to obtain the target image set. This ensures that the image data not only contains the original pixel information but also adds semantic descriptions. Through feature extraction, key features in the image can be identified, such as object edges, textures, and colors. These feature labels provide a richer information foundation for subsequent semantic segmentation and 3D modeling.

[0050] In one implementation, semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data. Based on this spatial point cloud data, a 3D target region is obtained by modeling the subject. This technique allows for the accurate extraction of 3D spatial information from 2D image data, constructing a 3D model of the target object. Semantic segmentation distinguishes between the subject and non-subject parts of the image, thus retaining only points related to the subject in the point cloud data, improving the accuracy of 3D modeling. For example, in robotic harvesting tasks, accurate 3D target region modeling helps the robot accurately identify and locate the target fruit, providing accurate 3D positional information for subsequent harvesting operations.

[0051] In one implementation, a final pose observation matrix is ​​generated based on the 3D target region, and the camera position is updated according to this matrix. This technique allows the camera's position and observation angle to be dynamically adjusted based on the 3D model of the target object, thereby achieving optimal observation results. The final pose observation matrix ensures that the camera observes the target object from the most suitable perspective and position, reducing occlusion and viewing angle deviation, and improving the visibility and detection accuracy of the target object. For example, in automated agricultural inspection, the optimized camera position can better capture the detailed features of the fruit, improving the accuracy of fruit detection and classification, and preventing the incorrect collection of immature fruit.

[0052] In one implementation, mapping semantic segmentation results to three-dimensional space effectively extracts fruit cluster point clouds, providing a more accurate ability to distinguish tomato clusters than two-dimensional images. This technique enables the system to analyze the spatial relationship between tomato clusters and adjacent stems from a three-dimensional perspective, accurately assessing the spatial extent of the clusters relative to the stems. This three-dimensional spatial analysis not only helps determine the optimal camera orientation but also generates suitable viewing poses for each tomato cluster. Generating viewing poses for multiple tomato clusters from a global perspective allows the robotic arm to perform harvesting operations efficiently. This method, starting from the perspective of the end effector camera, significantly improves the accuracy of picking point identification and positioning, providing more reliable technical support for automated harvesting.

[0053] In one embodiment, feature extraction of an image set to obtain feature labels includes:

[0054] Feature labels are obtained by extracting features from the image set using a pre-defined target model;

[0055] The target pre-defined model includes a backbone network and a neck structure; improvements of the target pre-defined model compared to the YOLOv8 model include:

[0056] Replace the Conv modules in layers 1, 3, 5, and 7 of the backbone network with the target Conv modules, and replace the C2f modules in layers 2, 4, 6, and 8 of the backbone network with the target C2f modules;

[0057] Replace each C2f module in the neck structure with the target C2f module, replace each Conv module in the neck structure with the target Conv module, add the target Conv module between the 12th and 13th layers, add the target Conv module between the 4th and 14th layers, add the target Conv module between the 6th and 11th layers, and add the target Conv module between the 9th and 10th layers.

[0058] A first attention module is added between layers 15 and 16 in the neck structure, a second attention module is added between layers 18 and 19 in the neck structure, and a third attention module is added after layer 21 in the neck structure to obtain the target preset model.

[0059] In one implementation, see [link to implementation details]. Figure 2 , Figure 2 This invention provides a schematic diagram of the model structure of a machine vision-based crop subject identification and localization method; Concat represents stitching, Upsample represents upsampling, SPPF is the spatial pyramid pooling module, Output represents output, and Conv module (0) is the input of the target model.

[0060] In one implementation, the first, second, and third attention modules are all SimAM (Simple Attention, which does not introduce additional parameters during training, and its fast computation speed makes it suitable for training with limited hardware resources). The processing results of the first, second, and third attention modules are the output of the neck structure, which is also the input of the detection head. The introduction of attention modules allows the model to focus more on important regions in the image, thereby improving feature representation and detection performance. By adding attention modules at key locations in the neck structure, the model can better capture the feature information of the target object while reducing attention to background noise, thus improving detection accuracy and robustness.

[0061] In one implementation, the improved model can process image information more efficiently, extract more representative and discriminative features, and strengthen important features through an attention mechanism, enabling the detection head to receive higher quality feature inputs. The above improvements enable the target preset model to achieve more accurate target detection while maintaining high computational efficiency, thus improving the performance and effectiveness of the model in practical applications.

[0062] In one embodiment, the working principle of the target Conv module includes:

[0063] Obtain the first original feature map, determine the number of channels in the original feature map, and perform channel separation on the original feature map according to preset rules to obtain the first branch map and the second branch map;

[0064] The first branch graph is input into the Conv module to obtain the third branch graph, and the second branch graph is input into the DWConv module to obtain the fourth branch graph;

[0065] The third and fourth branch graphs are concatenated to obtain an intermediate feature map, and a channel recombination operation is performed on the intermediate feature map to obtain the final feature map.

[0066] In one implementation, for example, the original feature map has C channels. The input feature map is first split into two parts according to the number of channels (each accounting for half of the total number of channels, namely the first branch map and the second branch map). They are processed by different convolution strategies and then complementary information is fused. For the feature map of one set of channels (the first branch map), standard convolution (Conv) is used. Standard convolution will pay attention to both spatial and channel correlations and extract relatively comprehensive features (to obtain the third branch map).

[0067] In one implementation, the feature maps of another set of channels (the second branch map) are processed using depthwise convolution (DWConv). DWConv first performs spatial convolution (the depthwise part, focusing on spatial details within each channel) channel-by-channel, capturing local features within each channel with relatively low computational cost. The output feature maps of the ordinary convolution branch (Conv) and the depthwise convolution branch (DWConv) are then concatenated (Concat) along the channel dimension. At this point, the total number of channels after concatenation is C, which is equivalent to initially fusing the complementary features extracted by the two convolutions.

[0068] In one implementation, the concatenated feature maps undergo a channel shuffle (recombination) operation. After processing the preceding branches, the channel features exhibit certain "grouping characteristics" (the feature distributions of ordinary convolution and depthwise convolution differ). Shuffling breaks this grouping, allowing channel features from different sources to be fully mixed, ensuring that subsequent layers can utilize all features in a balanced manner, avoiding information "isolation," and improving feature interactivity.

[0069] In one embodiment, the working principle of the target C2f module includes:

[0070] Obtain the second original feature map, input the second original feature map into the Conv module to obtain the first feature map, input the first feature map into the PConv module to obtain the second feature map, input the second feature map into the PConv module to obtain the third feature map, concatenate the first feature map, the second feature map and the third feature map to obtain the fourth feature map, and input the fourth feature map into the Conv module to obtain the fifth feature map.

[0071] In one implementation, the second original feature map is input into the Conv module to obtain the first feature map. The first feature map is then passed through two PConv modules to obtain the second and third feature maps respectively. Through the layer-by-layer extraction and refinement of features, the Conv module can extract basic feature information, while the PConv module further enhances the details and semantic information of the features.

[0072] In one implementation, the first, second, and third feature maps are concatenated to obtain a fourth feature map. This concatenation operation achieves the fusion of features at different levels. Since each feature map contains feature information from different stages, the fused fourth feature map integrates these different levels of features to obtain a comprehensive feature representation. This allows the model to utilize information from the original feature map to the feature map after multi-layer processing, enhancing the expressive power and diversity of the features. By integrating features at different levels, the model can better capture global and local information in the feature map, thereby improving the ability to recognize and understand targets.

[0073] In one implementation, the fused fourth feature map is input into the Conv module to obtain the fifth feature map. The fused features are then further optimized and adjusted. Through this convolutional operation, the model can reprocess the fused features, removing redundant information and highlighting important features, thus obtaining a more optimized and refined feature representation. The final fifth feature map, as the output of the C2f module, provides high-quality feature information for subsequent network layers, improving the overall network performance and accuracy.

[0074] In one embodiment, semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data, including:

[0075] A lightweight semantic segmentation network is used to extract the target images from the target image set to obtain the classification mask, and then point cloud spatial point cloud data is generated based on the classification mask.

[0076] The formula for point cloud computing is:

[0077] ;

[0078] Among them, (X) i ,Yi Z i Let (u) be the coordinates of the i-th point in three-dimensional space. i ,v i Let D be the pixel coordinates of pixel i. i is the pixel depth value, s is a constant scaling factor, (c x ,c y ,f x ,f y ) represents the camera's intrinsic parameters, M(u) i ,v i ) is a pixel (u i ,v i The mask pixels are 1 for the main subject and 2 for the non-main subject.

[0079] In one implementation, a classification mask (M(u)) generated by a lightweight semantic segmentation network is used. i ,v i This allows for precise differentiation between pixel regions of the tomato fruit (main subject, value 1) and the stem (non-main subject, value 2), avoiding background noise interference. Combined with the depth value D... i After being converted into a 3D point cloud, the spatial point cloud data of the target object (such as a cluster of tomatoes) can be extracted separately. Compared with using the full scene point cloud directly, it reduces the interference of irrelevant point clouds on subsequent clustering and pose estimation, making the spatial location (such as center coordinates and radius) of the tomato cluster more accurate.

[0080] In one implementation, the point cloud computing formula includes the camera's inherent parameters (c) x ,c y ,f x ,f y ) is obtained through camera calibration, where (c x ,c y (f) represents the coordinates of the optical center of the image. x ,f y (X) represents the focal length parameter, and a mapping relationship between 2D pixel coordinates and 3D space is established based on the pinhole imaging model. For example, in the formula (X) i ,Y i ) via depth Z i The calculations involving focal length and pixel offset convert pixel coordinates into physical coordinates in the camera coordinate system, ensuring that the spatial scale of the point cloud data is consistent with the actual scene.

[0081] In one implementation, point cloud computing is performed based on the classification mask. Using the above formula, the pixels in the image can be quickly converted into point cloud data in three-dimensional space, making the conversion process from two-dimensional image to three-dimensional point cloud more efficient and reducing computing time and resource consumption.

[0082] In one implementation, Xi ,Y i Z i : Represent the X, Y, and Z coordinates (Zi, Z ...) of the i-th point in three-dimensional space, respectively. i X represents the actual depth of the pixel. i ,Y i By using depth and camera intrinsics, pixel coordinates are mapped to XY coordinates in 3D space. These are the position coordinates of each point in the point cloud data in 3D space, used to construct 3D models of objects or scenes; i ,v i Pixel coordinates represent the position of each pixel in an image on a two-dimensional image plane. In image processing, pixel coordinates are a fundamental way of representing image data; they reflect the row and column positions of pixels in the image matrix. i : Pixel depth value, which is the depth information corresponding to each pixel in an image. The depth value is usually obtained by a depth sensor or by image processing algorithms (such as stereo vision matching algorithms). It provides the distance information of the pixel relative to the camera in three-dimensional space; s: Constant scaling factor, used to scale the depth value. Since the depth value obtained by the depth sensor may need to be adjusted according to the specific measurement range and unit, this scaling factor can adjust the scale of the depth value; c x c y These are camera intrinsic parameters, representing the X and Y coordinates of the camera's optical center in the image coordinate system. The optical center is an important parameter in the camera imaging system; it is related to the camera's optical structure and reflects the position of the origin of the image coordinate system. x f y These are camera intrinsic parameters, representing the pixel values ​​of the camera's focal length in the X and Y directions, respectively. Focal length is a key parameter of the camera's optical system, affecting the image size and shape. In point cloud computing, the focal length parameter is used to convert pixel coordinates into three-dimensional spatial coordinates; M(u i ,v i ): Mask pixels, used to distinguish different categories of pixels in an image. A value of 1 represents the subject, and a value of 2 represents a non-subject. The result is generated by a lightweight semantic segmentation network and used in the point cloud computing process to distinguish which pixels belong to the target object (subject, tomato) and which pixels belong to the background (non-subject, stem).

[0083] In one embodiment, modeling the subject based on spatial point cloud data to obtain a three-dimensional target region includes:

[0084] The main point cloud data is determined based on the spatial point cloud data. The main point cloud data is clustered to obtain a set of main clusters. The 3D modeling data is obtained by calculating each main cluster in the set of main clusters. The 3D target region is obtained by modeling based on the 3D modeling data. The set of main clusters contains multiple main clusters. The 3D modeling data includes the center coordinates and radius.

[0085] In one implementation, the formula for calculating the center coordinates is: Radius calculation formula: , where P center The three-dimensional center coordinates of the main body (X) center ,Y center Z center (Unit: mm; N is the number of point clouds in the main cluster, P) i Let X be the three-dimensional coordinates (X) of the i-th point within the cluster. i ,Y i Z i ), || is the Euclidean distance operator (to calculate the spatial distance from a point to the center), r is the radius of the bounding sphere of the tomato cluster (unit: mm, used to generate the cylindrical stem point cloud extraction region); the 3D modeling data is the modeling data of any subject cluster.

[0086] In one implementation, the main point cloud data is determined based on spatial point cloud data, and the main point cloud data is clustered to obtain a set of main clusters. This technique enables the accurate extraction of point cloud data of the target object from complex spatial point cloud data, providing a precise data foundation for subsequent 3D modeling. 3D modeling data, including center coordinates and radius, is calculated for each main cluster in the set of main clusters. This allows for the rapid calculation of key data for 3D modeling from the set of main clusters, improving the efficiency and accuracy of modeling.

[0087] In one embodiment, generating the final pose observation matrix based on the 3D target region includes:

[0088] The equations of non-subject lines are determined based on the three-dimensional target region. The starting point coordinates and non-subject direction vectors are then determined based on the equations of non-subject lines. The target center coordinates of the target subject cluster are obtained. The line parameters are calculated based on the starting point coordinates, target center coordinates, and non-subject direction vectors. The target subject cluster is any one of the subject cluster sets.

[0089] The coordinates of the perpendicular point are calculated based on the line parameters, the starting point coordinates, and the non-main direction vector. The coordinates of the perpendicular point and the target center coordinates are then projected onto the world coordinate system to obtain the initial direction vector.

[0090] The initial direction vector is used as the Z-axis direction vector. The upward direction vector is determined and used as the Y-axis direction vector. The X-axis direction vector is obtained by cross product of the Z-axis and Y-axis direction vectors. The initial pose matrix is ​​obtained by combining the X-axis, Z-axis, and Y-axis direction vectors with the target center coordinates. The initial pose matrix is ​​transformed according to the preset settings to obtain the final pose observation matrix.

[0091] In one implementation, by calculating the perpendicular point from the center of the tomato cluster to the straight line of the stem, the reference point for the observation direction can be accurately determined, providing an accurate reference point for subsequent observation and operation. As the point on the stem closest to the center of the tomato, the perpendicular point can effectively reduce detection errors caused by occlusion, improve the accuracy of observation pose estimation, provide a more reliable positioning basis for the operation of the robotic arm in complex environments, and ensure the accuracy and reliability of the operation.

[0092] In one implementation, the perpendicular point and the center of the tomato cluster are projected onto the XOY plane of the world coordinate system to generate an initial direction vector. An orthogonal basis (X, Y, and Z axis direction vectors) is then constructed through a cross product, ensuring that the observation direction is reasonably constrained to a horizontally reachable positive direction for the robotic arm, avoiding unreachable situations such as backward pose. By constructing the orthogonal basis, the rationality and reachability of the observation direction are ensured, improving the operational flexibility and adaptability of the robotic arm. This not only improves the reachability of the robotic arm but also reduces camera field-of-view obstruction caused by directional deviations, thus providing better visual conditions for subsequent recognition and operation.

[0093] In one implementation, the initial pose matrix is ​​converted into the final observation pose by rotating around the x-axis (pitch angle) and translating along the z-axis, causing the camera to tilt downwards and move closer to the tomato cluster. This technique effectively expands the camera's coverage of the tomato cluster and improves the camera's adaptability to different viewpoints. Combined with depth camera intrinsics, the spatial resolution of the point cloud data is improved, thus providing higher-quality data support for subsequent picking point detection. This adjustment allows the camera to better adapt to different observation scenarios, improving the camera's ability to detect target objects and its operational flexibility.

[0094] In one implementation, the equation of the non-principal straight line is: Where L(t) is the equation of a non-principal straight line, P l Let be the starting coordinates, and t be the line parameter. It is a non-subjective direction vector;

[0095] Linear parameters: , where P l Let P be the starting coordinate. center The coordinates of the target center are It is a non-subjective direction vector;

[0096] coordinates of the foot of the perpendicular: , where P D Let be the coordinates of the foot of the perpendicular, and t be the line parameter. It is a non-subjective direction vector; Where R is the rotation matrix, The X-axis direction vector (a horizontal orthogonal vector, derived from...) (Cross product) yields It is the Y-axis direction vector (upward direction vector, usually corresponding to the positive Y-axis or Z-axis of the world coordinate system (needs to be combined with hand-eye calibration)). The Z-axis direction vector (initial direction vector, which is also the observation direction vector, the horizontal projection vector from the center of the tomato cluster to the perpendicular point).

[0097] Initial pose matrix: , where T1 is the initial pose matrix (4×4 in dimension, including rotation and translation information), R is the rotation matrix, and P is the target center coordinates;

[0098] ;

[0099] Final pose observation matrix: Where T2 is the final pose observation matrix, and R x (β) is the matrix of rotation about the x-axis, where β is the rotation angle (unit: radians or degrees), and T1 is the initial pose matrix.

[0100] In one embodiment, the identification and positioning method is used to remove weeds from non-tomato plants.

[0101] The specific process of weed removal is as follows: the tomato plants are identified and located using identification and positioning methods;

[0102] Non-tomato plants are removed using a laser weeding device based on their labels.

[0103] In one implementation, a laser weeder, such as Carbon Robotics LaserWeeder G2 or Lightblade Smart Agriculture Intelligent Laser Weeding Robot, is used to remove weeds by setting a no-work zone within the positioning area of ​​the tomato plant tag (the laser weeder is prohibited from working in this area to prevent accidental damage to the tomato plants). In other words, all plants other than tomato plants are removed by using non-tomato plant tags.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for identifying and positioning a main body of a crop based on machine vision, characterized by, The method includes: A set of images of the target region is obtained, and feature labels are extracted from the image set to obtain feature labels. The feature labels are then added to the image set to obtain a target image set. Each image in the image set contains a crop as the main subject. The crop as the main subject is a tomato plant. The feature labels include: a tomato plant label and a non-tomato plant label. The target image set contains multiple target images. Semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data. Based on the spatial point cloud data, the subject is modeled to obtain a three-dimensional target region. The target image set contains multiple target images. A final pose observation matrix is ​​generated based on the three-dimensional target region, and the camera position is updated based on the final pose observation matrix. Feature extraction is performed on the image set to obtain feature labels, including: Feature labels are obtained by extracting features from the image set using a target preset model; The target preset model includes a backbone network and a neck structure; the improvements of the target preset model compared to the YOLOv8 model include: Replace the Conv modules in layers 1, 3, 5 and 7 of the backbone network with the target Conv modules, and replace the C2f modules in layers 2, 4 and 6 and 8 of the backbone network with the target C2f modules; Replace each C2f module in the neck structure with a target C2f module, replace each Conv module in the neck structure with a target Conv module, add a target Conv module between the 12th and 13th layers, add a target Conv module between the 4th and 14th layers, add a target Conv module between the 6th and 11th layers, and add a target Conv module between the 9th and 10th layers; A first attention module is added between the 15th and 16th layers of the neck structure, a second attention module is added between the 18th and 19th layers of the neck structure, and a third attention module is added after the 21st layer of the neck structure to obtain the target preset model. The working principle of the target Conv module includes: Obtain the first original feature map, determine the number of channels in the original feature map, and perform channel separation on the original feature map according to preset rules to obtain the first branch map and the second branch map; The first branch graph is input into the Conv module to obtain the third branch graph, and the second branch graph is input into the DWConv module to obtain the fourth branch graph; The third branch map and the fourth branch map are spliced ​​together to obtain an intermediate feature map, and a channel recombination operation is performed on the intermediate feature map to obtain the final feature map; The working principle of the target C2f module includes: Obtain a second original feature map, input the second original feature map into the Conv module to obtain a first feature map, input the first feature map into the PConv module to obtain a second feature map, input the second feature map into the PConv module to obtain a third feature map, concatenate the first feature map, the second feature map and the third feature map to obtain a fourth feature map, and input the fourth feature map into the Conv module to obtain a fifth feature map.

2. The method of claim 1, wherein, Semantic segmentation is performed on each target image in the target image set to obtain spatial point cloud data, including: A classification mask is obtained by extracting target images from the target image set using a lightweight semantic segmentation network, and point cloud spatial point cloud data is generated based on the classification mask. The formula for point cloud computing is: ； Among them, (X) i ,Y i Z i Let (u) be the coordinates of the i-th point in three-dimensional space. i ,v i Let D be the pixel coordinates of pixel i. i is the pixel depth value, s is a constant scaling factor, (c x ,c y ,f x ,f y ) represents the camera's intrinsic parameters, M(u) i ,v i ) is a pixel (u i ,v i The mask pixels are 1 for the main subject and 2 for the non-main subject. 3.The method of claim 1, wherein, The three-dimensional target region is obtained by modeling the subject based on the spatial point cloud data, including: Based on the spatial point cloud data, the main point cloud data is determined, and the main point cloud data is clustered to obtain a set of main clusters. The three-dimensional modeling data is calculated for each main cluster in the set of main clusters, and the three-dimensional target region is obtained by modeling based on the three-dimensional modeling data. The set of main clusters contains multiple main clusters. The three-dimensional modeling data includes center coordinates and radius.

4. The method of claim 3, wherein the method further comprises: The final pose observation matrix is ​​generated based on the three-dimensional target region, including: The equations of non-subject lines are determined based on the three-dimensional target region. The starting point coordinates and non-subject direction vectors are determined based on the equations of non-subject lines. The target center coordinates of the target subject cluster are obtained. The line parameters are calculated based on the starting point coordinates, the target center coordinates, and the non-subject direction vectors. The target subject cluster is any one of the subject cluster sets. The perpendicular point coordinates are calculated based on the line parameters, the starting point coordinates, and the non-subject direction vector. The perpendicular point coordinates and the target center coordinates are then projected onto the world coordinate system to obtain the initial direction vector. The initial direction vector is used as the Z-axis direction vector to determine the upward direction vector, and the upward direction vector is used as the Y-axis direction vector. The X-axis direction vector is obtained by performing a cross product of the Z-axis direction vector and the Y-axis direction vector. The initial pose matrix is ​​obtained by combining the X-axis direction vector, the Z-axis direction vector, the Y-axis direction vector, and the target center coordinates. The initial pose matrix is ​​then transformed according to preset settings to obtain the final pose observation matrix.

5. The method of claim 1-4, wherein, This identification and positioning method is used to remove weeds from non-tomato plants; The specific process of removing impurities is as follows: the tomato plants are identified and located using an identification and positioning method; The non-tomato plants are removed using a laser weeding device based on the labels on the non-tomato plants.

Images