An image-based irregular strip object detection and reconstruction method

Abstract

This invention discloses an image-based method for detecting and reconstructing irregular strip-shaped objects, comprising the following steps: S1: image acquisition; S2: image preprocessing; S3: optimizing the network model; S4: setting network model parameters; S5: inputting the training set into the optimized network model for training, saving the trained weight file, and using the test set to test the model's segmentation effect on the target; S6: reconstructing branch segments. This method can effectively detect and segment branches with similar background colors, overlaps, and occlusions, without requiring extensive data statistics, and achieves high reconstruction accuracy.

Description

Technical Field

[0001] This invention relates to the field of instance segmentation technology, specifically a method for detecting and reconstructing irregular strip-shaped objects based on images. Background Technology

[0002] Strip-shaped object detection and reconstruction technology has significant practical value in many fields. This technology typically employs various techniques, such as laser scanning, structured light, or depth cameras, to capture the geometric information of objects and uses image processing and 3D reconstruction methods to generate 2D and 3D models. These methods aim to detect and reconstruct the shape of strip-shaped objects in images or videos, usually to improve the accuracy of object recognition and tracking in real-world situations. For example, in agricultural automation applications, this technology is used to identify branches, assisting in the automation and intelligentization of operations such as fruit tree pruning and fruit harvesting. However, this field still needs continuous improvement and expansion in terms of technical precision, accuracy, and efficiency, especially in complex field or orchard environments.

[0003] Early work on strip object detection and reconstruction primarily relied on machine learning algorithms. These methods can be categorized into two types based on data type: one uses LiDAR to acquire point cloud data and performs detection and reconstruction based on 3D information. For example, it utilizes label-related topological structures and geometric priors to model the skeleton of the strip object, extracting and fitting object curves from the point cloud data using graphical manipulation. The main drawback of this type is the need for high-quality LiDAR point clouds, which are often affected by lighting conditions in real-world environments, reducing the effectiveness of object detection and reconstruction. The other method is based on RGB images, extracting object features from the image using color space transformation, thresholding and clustering segmentation, and edge detection for segmentation and reconstruction. Some researchers have proposed a Hough transform-based strip object detection algorithm based on gradient phase grouping, achieving strip object detection by quickly detecting straight lines. Others have proposed an iterative thresholding segmentation method based on contrast-limited adaptive histogram equalization. However, due to the discontinuities in the segmented objects, it fails to obtain complete object information. Additionally, some researchers have used a method combining detectors and preprocessing techniques to detect objects using depth information. This method uses color space combination and segmentation to eliminate background information in the image, and combines depth information to further improve the detection rate. Although the above method can extract strip-shaped objects from images, the background is relatively simple, and in unstructured environments, it is easily affected by factors such as changes in lighting and occlusion, which can lead to a decrease in recognition progress.

[0004] With the rapid development of deep neural networks in object recognition, researchers have begun to combine traditional vision techniques with deep learning for the detection of strip-shaped objects. Compared to the traditional methods mentioned above, the powerful feature extraction and self-learning capabilities of deep neural networks can better adapt to complex backgrounds. For example, SegNet semantic segmentation network can be used to segment strip-shaped objects with common appearance and features, but this method is only suitable for objects with simple backgrounds. Some researchers use improved convolutional neural networks (R-CNN) to detect strip-shaped objects, while others use improved semantic segmentation networks to segment strip-shaped objects and utilize curve fitting techniques to model the objects. Still others use discretization labeling methods to label data and use Mask R-CNN to detect and segment strip-shaped objects, proposing a multi-constraint method to reconstruct complete branches for the segmented branches. Furthermore, in another study, researchers used CNN to coarsely detect branches and combined image processing and multinomial fitting methods to accurately segment and reconstruct the branches, ultimately achieving a good average reconstruction accuracy. However, this approach is relatively simple and cannot be directly applied to long, thin, and curved strip-shaped objects.

[0005] Most research focuses on specific categories of strip-shaped objects with relatively simple backgrounds and structures, and the methods used lack universality. While some progress has been made in detecting strip-shaped objects, it is not effective in identifying curved and irregular strip-shaped objects against similar colored backgrounds. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an image-based method for detecting and reconstructing irregular strip-shaped objects. This method can effectively detect and segment branches even with similar background colors, overlaps, and occlusions, without requiring extensive data statistical work, and achieves high reconstruction accuracy.

[0007] The technical solution to achieve the objective of this invention is:

[0008] An image-based method for detecting and reconstructing irregular strip-shaped objects includes the following steps:

[0009] S1: Image Acquisition: The user uses an image acquisition device to acquire images of the irregular strip-shaped object, and names the acquired images according to the COCO dataset format, while creating three folders named Annotations, TrainImage, and ValImage.

[0010] S2: Image preprocessing:

[0011] S2-1, Image Labeling: For the image acquired in step S1, use the Labelme image labeling tool to discretize and label the irregular strip-shaped objects in the image, marking the outline and category of the objects;

[0012] S2-2, Image Augmentation: Augment the images acquired in step S1 by selecting different image augmentation methods and dividing the image into training and test sets.

[0013] S3: Optimize the network model: The feature extraction part of the original network uses ResNet101 supplemented by Feature Pyramid Network (FPN) as the backbone network. Deformable convolution is used to extract strong semantic information in the image. By adding an offset to each sampling point in the convolution kernel, the receptive field can be automatically adjusted to adapt to the shape of the object. Then, by adding a parallel standard convolution, the bias of the convolution kernel is learned. The offset is added to the convolution kernel so that the sampling point position of the convolution kernel on the feature map changes adaptively with the image content, thereby improving the generalization ability of objects of different shapes.

[0014] S4: Set network model parameters: Based on the computer configuration used in the experiment and the required training effect, set the size of the input image, the number of segmentation types, and the number of iterations in the configuration file of the Mask RCNN network model.

[0015] S5: Input the training set into the optimized network model for training, save the weight file obtained from training, and use the test set to test the model's segmentation effect on the target;

[0016] S6: Reconstruct branch segments:

[0017] S6-1: Image segmentation preprocessing: The segmented mask generated after the input image is segmented by the model may have the problem of repetition. In this case, using the branch reconstruction algorithm will cause branch reconstruction errors. In order to reduce the occurrence of errors, the intersection-union ratio between the extracted minimum bounding rectangles is calculated, and the minimum bounding rectangles with the intersection-union ratio greater than the threshold are discarded.

[0018] S6-2: Initial refactoring, the steps are as follows:

[0019] S6-2-1: Constructing a segmented model: For the torso of a strip-shaped object, hereinafter referred to as a branch segment, a branch segment model is used. To simplify the representation of the minimum bounding rectangle of the preprocessed mask, where Let represent the set of points on the length of the branch segment. For the initial branch segment, It contains only 3 point elements. The four corner points of the minimum bounding rectangle, starting clockwise from the top left corner, are used to determine the boundary of the sector search. For bifurcation segments, the center point of the minimum bounding rectangle is used.

[0020] S6-2-2: Constructing the search region: Relative to the current segmented branch, the strip-shaped object always extends at its two ends, i.e., above and below or left and right. Therefore, for each branch segment, a bidirectional fan-shaped search region is constructed along its long axis in the simplified branch model.

[0021] S6-2-3: Search for all other branch segments within the target branch segment search area in the search image, defining the fan angle of the search area. and And the offset angles of the points constituting other branch segments relative to the target branch segment. and Specifically, as shown in formulas (1) and (2):

[0022] (1)

[0023] (2)

[0024] in Indicates the major axis points that constitute the other branch segment models. Represents the total number of points in the target branch segment model, when When the target branch segment is in the target branch segment, temporarily mark the branch segment at the top of the search area; when When this happens, the branch segment is temporarily marked in the search area at the tail of the target branch segment;

[0025] S6-2-4: Reverse Search: In order to accurately divide the branch segments and reduce the influence of adjacent branches, a strategy of reverse search for the target branch segment is adopted for the marked branch segments. At the same time, the offset angle of the two is constrained by formula (3) to further ensure that they are on the same branch as the target branch segment. or For a bifurcation segment, when its simplified center point falls within the search area, it is considered to be on the same branch as the target branch segment.

[0026] (3)

[0027] S6-2-5: Constraining the search depth: Formula (4) constrains the depth of the search range. Considering the problems of occlusion and missed identification, the depths of the search branch segments and forks are set as follows: and Within the search range, calculate the distance between the target branch segment and other branch segments and forks within the search area. If the closest one is a fork segment, merge it with the target branch segment and stop the search connection. If the closest one is a branch segment, merge it with the target branch segment and update it. Repeat the above search process for the updated target branch segment until it is connected to a fork segment or there are no other branch segments belonging to the same branch within the search area.

[0028] (4)

[0029] S6-3: Secondary Reconstruction: After the initial reconstruction in S6-2, individual branch segments and forks are connected to the same branch. However, due to the problems of missed detection and occlusion of irregular strip-shaped objects and their connected side branches, the same complete branch may be divided into multiple segments during connection. Therefore, secondary connections are performed on other segments belonging to the same branch. For multiple branches with the same fork point, the branch with the fork point as the tail node is defined as the main branch, and the branch with the fork point as the top node is defined as the secondary branch. The relative angle between the main branch and each secondary branch is calculated. ,like Since missed detections would result in only one pair of main branches and secondary branches not belonging to the same branch, a threshold is set to constrain the secondary branches. Because the relative angle between two segments of the same branch tends to 180°, considering the varying degrees of curvature of the strip-shaped object, the threshold is set to 100°, thus connecting it to the main branch. For multiple branches without the same bifurcation point, the same search and connection method as S6-2 is used. At this point, the angle constraint of the sector search area is... and Within the range, distance constraints are Within the range, the deflection angle is constrained. This is to reduce the impact of parallel branches on the reconstruction.

[0030] This technical solution further improves upon the Mask R-CNN algorithm and proposes a branch segment reconstruction method based on bidirectional sector search. The idea is to first optimize the convolution operation by replacing the original ordinary convolution with deformable convolution to improve the segmentation effect on irregular objects. Based on the segmentation result of the algorithm, the result image is preprocessed, and a branch segment reconstruction method is proposed according to the extension trend of irregular strip-shaped objects to reconstruct the segmented branches and obtain complete information on irregular strip-shaped objects.

[0031] This technical solution optimizes the Mask R-CNN network model structure by adding deformable convolutions, which enables better segmentation of irregular strip-shaped objects. Furthermore, it extracts complete branch information by reconstructing the segmented branch segments using the proposed bidirectional sector search method.

[0032] Compared with existing technologies, this technical solution has the following advantages:

[0033] (1) The optimized network model can effectively detect and segment branches in cases of similar background color, overlap, and occlusion;

[0034] (2) The proposed branch segment reconstruction method does not require a lot of data statistics work, and compared with previous methods, it can still maintain a high reconstruction accuracy in relatively complex backgrounds. Attached Figure Description

[0035] Figure 1 Flowchart for an embodiment;

[0036] Figure 2 This is a diagram illustrating the critical case where the minimum intersection-union ratio satisfies the reconstruction algorithm in the embodiments.

[0037] Figure 3 A search area map is defined for this embodiment;

[0038] Figure 4 This is a diagram illustrating the branch segment search process in the embodiment;

[0039] Figure 5 This is a diagram showing the deflection angle constraint in the embodiment;

[0040] Figure 6 The image shows the segmentation effect of the optimized Mask RCNN network model on passion fruit tree branches in the example.

[0041] Figure 7 The image shows the reconstruction effect of an example. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0043] Example:

[0044] This example uses passion fruit tree branches.

[0045] Reference Figure 1 An image-based method for detecting and reconstructing irregular strip-shaped objects includes the following steps:

[0046] S1. Image Acquisition: Users use a digital camera or other image acquisition device to acquire images of passion fruit tree branches, and name the acquired images according to the COCO dataset format, while creating three folders named Annotations, TrainImage, and ValImage; In this example, a digital camera is used to acquire images of passion fruit tree branches.

[0047] S2. Image preprocessing:

[0048] S2-1, Image Labeling: For the image acquired in step S1, use the Labelme image labeling tool to discretize and label the branches in the image, marking the outline and category (branching and forking) of the branches.

[0049] S2-2, Image Augmentation: The images acquired in step S1 are augmented. Augmentation processing includes adding noise, blurring, and adjusting contrast. Different image augmentation methods are selected for augmentation, and training and test sets are divided with a ratio of 7:3.

[0050] S3. Optimize the network model: The feature extraction part of the original network uses ResNet101 supplemented by Feature Pyramid Network (FPN) as the backbone network to extract strong semantic information in the image. Deformable convolution is used. By adding an offset to each sampling point in the convolution kernel, the receptive field can be automatically adjusted to adapt to the shape of the object. By adding a parallel standard convolution, the bias of the convolution kernel is learned. The offset is added to the convolution kernel so that the sampling point position of the convolution kernel on the feature map changes adaptively with the image content, thereby improving the generalization ability of objects of different shapes.

[0051] S4. Set network model parameters: The computer configuration used in this experiment is: Intel Xeon Gold 6230 CPU, NVIDIA Tesla V100 GPU and Ubuntu 18.04 operating system. Based on these configurations and the required training effect, set the size of the input image, the number of segmentation types, the number of iterations, etc. in the configuration file of the Mask RCNN network model.

[0052] S5. Input the training set into the optimized network model for training, save the weight file obtained from training, and use the test set to test the model's segmentation effect on the target;

[0053] S6: Refactor the branch segment:

[0054] S6-1: Image Segmentation Preprocessing: Sometimes, the branch segment masks generated after the input image is segmented by the model have duplicates. In such cases, using a branch reconstruction algorithm will lead to branch reconstruction errors. To reduce these errors, the intersection-union ratio (IUR) of the extracted minimum bounding rectangles is calculated, and minimum bounding rectangles with IUR greater than a threshold are discarded. During the dataset labeling stage, the aspect ratio of the labeled quadrilaterals is set to be within the range of 2-4. Therefore, the aspect ratio of the minimum bounding rectangles of the extracted segmentation masks is also approximately within the range of 2-4. Figure 2 The figure shows the critical case for the minimum crossover-union ratio (CUI) when the reconstruction algorithm is satisfied. The CUI is 0.2 at this time. Considering that the minimum enclosing rectangle has a deviation in the minor axis direction, the threshold is set to 0.15.

[0055] S6-2: Initial refactoring, the steps are as follows:

[0056] S6-2-1: Constructing a branch segment model: For the branch part, use a branch segment model. To simplify the representation of the minimum bounding rectangle of the preprocessed mask, where Let represent the set of points on the length of the branch segment. For the initial branch segment, It contains only 3 point elements. The four corner points of the minimum bounding rectangle, starting clockwise from the top left corner, are used to determine the boundary of the sector search. For the bifurcation part, the center point of the minimum bounding rectangle is used.

[0057] S6-2-2: Constructing the Search Region: Relative to the current segmented branch, the branch always extends and grows at both ends (above and below, or left and right). Therefore, for each branch segment, a bidirectional fan-shaped search region is constructed along its long axis based on the simplified branch model, such as... Figure 3 As shown, the dashed line and The boundary of the top fan-shaped search area is limited by the dashed line. and The boundaries of the tail sector search area were limited. );

[0058] S6-2-3: Search for all other branch segments within the target branch segment search area in the search image, defining the fan angle of the search area. and And the offset angles of the points constituting other branch segments relative to the target branch segment. and Specifically, as shown in formulas (1) and (2):

[0059] (1)

[0060] (2)

[0061] in Indicates the major axis points that constitute the other branch segment models. Represents the total number of points in the target branch segment model, when When the target branch segment is in the target branch segment, temporarily mark the branch segment at the top of the search area; when When this happens, the branch segment is temporarily marked in the search area at the tail of the target branch segment;

[0062] S6-2-4: Reverse Search: In order to more accurately divide the branch segments and reduce the influence of adjacent branches, a strategy of reverse search for the target branch segment is adopted for the marked branch segments. At the same time, the offset angle of the two is constrained by formula (3) to further ensure that they are on the same branch as the target branch segment. or For a bifurcation segment, when its simplified center point falls within the search area, I consider it to be on the same branch as the target branch segment. The entire branch segment search process is as follows: Figure 4 As shown;

[0063] (3)

[0064] S6-2-5: Constraining the search depth: Formula (4) constrains the depth of the search range. Considering the problems of occlusion and missed identification, after multiple experiments, this example sets the depth of the search branch segment and the fork segment to be respectively. and Within the search range, calculate the distance between the target branch segment and other branch segments and forks within the search area. If the closest one is a fork segment, merge it with the target branch segment and stop the search connection. If the closest one is a branch segment, merge it with the target branch segment and update it. Repeat the above search process for the updated target branch segment until it is connected to a fork segment or there are no other branch segments belonging to the same branch within the search area.

[0065] (4)

[0066] S6-3: Secondary Reconstruction: After the initial reconstruction in S6-2, individual branch segments and forks are connected to the same branch. However, due to the presence of lateral branches and issues such as missed detections and occlusions, the same complete branch may be divided into multiple segments during connection. Therefore, secondary connections are performed on other segments belonging to the same branch. For multiple branches with the same fork point, the branch with the fork point as the tail node is defined as the main branch, and the branch with the fork point as the apex node is defined as the secondary branch. The relative angle between the main branch and each secondary branch is calculated. ,like (Since missed detections may result in only one pair of main branches and secondary branches that do not belong to the same branch, a threshold is set to constrain the secondary branches. Because the relative angle between two segments of the same branch tends to 180°, considering the different degrees of bending of the branches, the threshold is set to 100°.) Then it is connected to the main branch. For multiple branches that do not have the same branching point, the same search and connection method as S6-2 is used. At this time, the angle constraint of the sector search area is... and Within the range, distance constraints are Within the range, such as Figure 5 The deflection angle is constrained as shown. This is to reduce the impact of parallel branches on the reconstruction.

[0067] This example demonstrates the effect of using an optimized Mask R-CNN network model to segment passion fruit tree branches, as shown below. Figure 6 As shown, the reconstruction effect is as follows Figure 7As shown in the figure, compared with existing methods, the method in this example can better detect and reconstruct strip-shaped objects with occlusion, loss, and curvature against a background of similar color.

Claims

1. An image-based method of detecting and reconstructing irregular strip objects, characterized by, Comprise the following steps: S1: image acquisition: the user uses the image acquisition device to collect the image of the irregular strip object, and names the collected image according to the format of COCO data set, and creates three folders named Annotations, TrainImage and ValImage; S2: image preprocessing: S2-1, image labeling: for the image collected in step S1, the Labelme image labeling tool is used to discretely label the irregular strip object in the image, and the outline and category of the object are labeled; S2-2, image augmentation: the image collected in step S1 is augmented, different image augmentation methods are selected for augmentation and division of training set and test set; S3: optimization of network model: the feature extraction part of the original network adopts ResNet101 assisted by feature pyramid network as the backbone network to extract strong semantic information in the image, uses deformable convolution, adds an offset to each sampling point in the convolution kernel to make the receptive field automatically adjust to adapt to the shape of the object, and then adds a parallel standard convolution to learn the deviation of the convolution kernel, adds the offset to the convolution kernel to make the sampling point position of the convolution kernel on the feature map change adaptively with the image content, so as to improve the generalization ability of different shaped objects; S4: setting network model parameters: according to the computer configuration used in the experiment and the required training effect, in the configuration file config of Mask RCNN network model, the size of model input image, the number of segmentation categories and the number of iterations are set; S5: input the training set into the optimized network model for training, save the weight file obtained by training, and test the segmentation effect of the model on the target using the test set; S6: restructure the branch segment: S6-1: segmentation image preprocessing: the segmentation mask generated after inputting the image into the model will have the problem of repetition, and using branch reconstruction algorithm in this case will cause branch reconstruction error, in order to reduce the occurrence of error, calculate the intersection over union between the extracted minimum bounding rectangles, and discard the minimum bounding rectangles with intersection over union greater than the threshold value; S6-2: initial reconstruction, the steps are as follows: S6-2-1: Constructing a segment model: for the trunk of a strip object, hereinafter referred to as a branch segment, a branch segment model is used to simplify the representation of the pre-processed mask represents the minimum bounding rectangle of the pre-processed mask, wherein represents the set of points on the branch segment long axis, for the initial branch segment, contains only 3 point elements, represents the four corner points of the minimum bounding rectangle starting from the upper left corner clockwise, used to determine the sector search boundary, for the branch segment, represented by the center point of the minimum bounding rectangle; S6-2-2: search area construction: relative to the current segmentation branch, the strip object always extends at both ends, that is, above and below or left and right, therefore, for each branch segment, a bidirectional fan-shaped search area is constructed along the long axis direction of the simplified branch model; S6-2-3: search for all other branch segments in the image that are located within the target branch segment search region, define the sector angle of the search region and and the offset angle of the points constituting the other branch segments relative to the target branch segment and , as shown in particular in equations (1) and (2): (1)， (2)， wherein represents a long axis point constituting the other branch segment model, represents the total number of point sets of the target branch segment model, when the branch segment is temporarily marked in the top search area of the target branch segment; when the branch segment is temporarily marked in the tail search area of the target branch segment; S6-2-4: Reverse search: In order to precisely divide the branch segment and reduce the influence of adjacent branches, the reverse search strategy is adopted for the marked branch segment to the target branch segment, and formula (3) is used to constrain the offset angle of the two to further ensure that they are on the same branch. At this time Or For the bifurcation segment, when the center point of its simplified falls within the search area, it is determined that it is on the same branch as the target branch segment. （3）； S6-2-5: Restrict the depth of search: formula (4) restricts the depth of search range, considering the occlusion and missed recognition problems, the depth of search branch segment and bifurcation segment is set to and The distance between the target branch segment and other branch segments and bifurcation segments in the search area is calculated in the search range, if the nearest distance is the bifurcation segment, it is merged with the target branch segment and the search connection is stopped, if the nearest distance is the branch segment, it is merged with the target branch segment and updated, and the above search process is repeated for the updated target branch segment until it is connected to the bifurcation segment or there is no other branch segment belonging to the same branch in the search area; (4)； S6-3: Secondary reconstruction: After the primary reconstruction of S6-2, the individual branch segments and bifurcation segments are connected to the same branch, but due to the problem of missing detection and occlusion of irregular strip objects and their connected side branches, the same complete branch will be segmented into multiple segments during connection, so the secondary connection is performed on other segments belonging to the same branch. For multiple branch segments with the same bifurcation point, the branch with the bifurcation point as the tail node is defined as the main branch, and the branch with the bifurcation point as the top node is defined as the secondary branch. The relative angle between the main branch and each secondary branch is calculated , if , due to the missing detection, there is only one pair of main branch and secondary branch that do not belong to the same branch, so a threshold is set to constrain the secondary branch. Because the relative angle of two segments of the same branch tends to 180°, considering the different degrees of bending of the strip object, the threshold is set to 100°, and it is connected with the main branch. For multiple branch segments without the same bifurcation point, the same search and connection method of S6-2 is used. At this time, the angle constraint of the fan-shaped search area is in the range of and , the distance constraint is in the range of , and the deflection angle is constrained to reduce the influence of parallel branches on reconstruction.

Images