Linear object segmentation method, device, equipment, medium and program product
Through the adaptive curvature convolution network and feature pyramid network, feature extraction and segmentation of linear objects is solved, and the segmentation difficulties caused by complex shapes and occlusion of linear objects are achieved, high-precision linear object segmentation is improved, and assembly efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202411082542.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-15
AI Technical Summary
The complex shape of linear objects and multiple cross-occlusions lead to problems of segmentation difficulties and errors. The prior art is difficult to accurately identify and locate linear objects, resulting in low assembly efficiency and accuracy.
Adaptive curvature convolution network is used to extract the target image feature, obtain multi-scale feature maps, and select linear object areas through candidate boxes to generate segmentation prediction results. Use feature pyramid network and segmentation mask prediction network to perform segmentation mask Generate and optimize, remove noise and artifacts, and realize binarization processing.
It improves the accuracy and efficiency of linear object segmentation, dynamically adapts to complex structures, improves the accuracy and topological continuity of segmentation results, and solves the problems of segmentation difficulties and errors.
Smart Images

Figure CN118968065A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and in particular to a linear object segmentation method, device, equipment, medium and program product. Background Art
[0002] Linear objects include wires, pipes, cables, ropes and other objects, which are widely present in industrial scenarios. For the automated assembly of linear objects, accurate segmentation can help robots accurately identify and locate linear objects, thereby improving assembly efficiency and accuracy. At the same time, accurate segmentation can help robots avoid or properly handle linear objects such as cables and pipes, avoiding damage and accidents during the assembly process.
[0003] However, due to the complex and changeable shapes of linear objects, their geometric shapes are uniform and complex, and their shapes vary from different perspectives. When multiple linear objects are stacked on each other, there are multiple intersections and complex occlusion relationships, which further aggravate the morphological changes of the target objects. This will lead to difficulties in linear object segmentation and segmentation errors. Summary of the invention
[0004] The technical solution of the present invention aims to provide a linear object segmentation method, device, equipment, medium and program product, which are used to solve the problems of linear object segmentation difficulty and segmentation error in the prior art.
[0005] To achieve the above object, the present invention is achieved by:
[0006] In a first aspect, an embodiment of the present invention provides a linear object segmentation method, comprising:
[0007] The linear objects in the target image are extracted by using an adaptive curvature convolutional network to obtain image features at least at two scales.
[0008] Performing feature fusion on the image features at the at least two scales to obtain a multi-scale feature map;
[0009] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a regional feature map;
[0010] According to the regional feature map, a segmentation prediction result of the linear object is obtained.
[0011] Optionally, the linear object segmentation method, wherein the feature extraction of the linear object in the target image is performed by an adaptive curvature convolutional network to obtain image features at least at two scales, comprises:
[0012] Extract features of linear objects in the target image according to the feature pyramid network to obtain image features at least at two scales;
[0013] The feature pyramid network includes at least two adaptive curvature convolution networks, one adaptive curvature convolution network is used to obtain image features of a scale, and the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network.
[0014] Optionally, in the linear object segmentation method, the adaptive curvature convolution network is used to deform the sampling points of the target image according to the offset so that the shape composed of the deformed sampling points conforms to the shape of the linear object.
[0015] Optionally, the linear object segmentation method further comprises:
[0016] The central sampling point of the target image is used as the offset starting point, and the offset of each sampling point is obtained according to the iteration direction and the offset rule, wherein the offset rule is used to indicate that the offset difference between adjacent sampling points is less than a preset threshold.
[0017] Optionally, the linear object segmentation method, wherein the linear object region is selected from the multi-scale feature map using a candidate frame to obtain a region feature map, comprises:
[0018] Selecting a linear object region in the target image to generate a candidate frame;
[0019] Aligning a candidate box in the target image with the multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map;
[0020] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a region of interest;
[0021] The region of interest is processed into a preset size to obtain a regional feature map.
[0022] Optionally, the linear object segmentation method, wherein obtaining the segmentation prediction result of the linear object according to the regional feature map, includes:
[0023] Generating a segmentation mask of the linear object according to the region feature map and the segmentation mask prediction network;
[0024] Removing noise and artifacts from the segmentation mask to obtain a segmentation mask after the removal process;
[0025] The segmentation mask after the removal process is binarized to obtain the segmentation prediction result of the linear object.
[0026] Optionally, in the linear object segmentation method, the segmentation prediction result includes at least one of the following:
[0027] the name of the linear object;
[0028] Bounding box of linear object area;
[0029] The pixel coordinates of the linear object.
[0030] Optionally, the linear object segmentation method further comprises:
[0031] Calculate the loss function according to the segmentation prediction result and the segmentation annotation result;
[0032] Back-propagating and updating the network parameters of the segmentation mask prediction network according to the loss function;
[0033] The loss function includes at least one of the following:
[0034] Loss function corresponding to the classification task;
[0035] Loss function for bounding box selection task;
[0036] Loss function corresponding to the segmentation task.
[0037] Optionally, in the linear object segmentation method, the loss function corresponding to the segmentation task includes a continuous homology loss function, and the continuous homology loss function is used to optimize the segmentation prediction results of the areas with breaks and incorrect connections.
[0038] In a second aspect, an embodiment of the present invention further provides a linear object segmentation device, comprising:
[0039] An extraction module, used for extracting features of linear objects in a target image through an adaptive curvature convolutional network to obtain image features at least at two scales;
[0040] A fusion module, performing feature fusion on the image features of the at least two scales to obtain a multi-scale feature map;
[0041] A selection module, used to select a linear object region from the multi-scale feature map using a candidate frame to obtain a regional feature map;
[0042] A prediction module is used to obtain a segmentation prediction result of the linear object according to the regional feature map.
[0043] In a third aspect, an embodiment of the present invention further provides a linear object segmentation device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the linear object segmentation method as described in the first aspect is implemented.
[0044] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium having a program stored thereon, and when the program is executed by a processor, the linear object segmentation method as described in the first aspect is implemented.
[0045] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the linear object segmentation method as described in the first aspect.
[0046] The beneficial effects of the above technical solution of the present invention are as follows:
[0047] The linear object segmentation method described in the embodiment of the present invention is used to extract features of linear objects in the target image through an adaptive curvature convolutional network to obtain image features of at least two scales; feature fusion is performed on the image features of at least two scales to obtain a multi-scale feature map; a linear object region is selected from the multi-scale feature map using a candidate frame to obtain a regional feature map; and a segmentation prediction result of the linear object is obtained based on the regional feature map. In this way, the use of an adaptive curvature convolutional network can dynamically adapt to complex and changeable linear object structures, extract key image features, thereby improving the efficiency and accuracy of image feature extraction, thereby improving the accuracy of segmentation results, and solving the problem of difficulty and error in linear object segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A schematic diagram of a flow chart of a linear object segmentation method according to an embodiment of the present invention;
[0049] Figure 2 A schematic diagram of a flow chart of one implementation of the method described in an embodiment of the present invention;
[0050] Figure 3 A schematic diagram of another implementation of the method according to an embodiment of the present invention;
[0051] Figure 4 A schematic diagram of another implementation of the method according to an embodiment of the present invention;
[0052] Figure 5 is a structural block diagram of a linear object segmentation device according to an embodiment of the present invention;
[0053] Figure 6 Schematic diagram of the hardware structure of the linear object segmentation device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0055] The embodiments of the present invention provide a linear object segmentation method, device, equipment, medium and program product to solve the problems of linear object segmentation difficulty and segmentation error in the prior art.
[0056] like Figure 1 As shown, an embodiment of the present invention provides a linear object segmentation method, comprising:
[0057] Step 101: extract features of linear objects in a target image through an adaptive curvature convolutional network to obtain image features at least at two scales.
[0058] Here, the adaptive curvature convolutional network can be called a deformable convolutional network, and image features can be better extracted through the adaptive curvature convolutional network.
[0059] In an embodiment of the present invention, optionally, an image set is obtained by collecting images of various linear objects in various occlusion situations in an industrial scene, and a target image is obtained from the image set. The linear objects may include pipelines and cables; and the various occlusion situations may include no occlusion, simple occlusion, and complex occlusion.
[0060] Here, the sizes of the various linear objects collected can be different, for example, ranging from 0.3 meters to 0.7 meters, and the materials of the various linear objects can also be different, such as reflective materials or non-reflective materials. The resolution of the collected image is 720P.
[0061] Optionally, before step 101, the method further includes:
[0062] The target image is preprocessed to obtain a preprocessed target image, wherein the preprocessing includes resizing and / or normalization. It should be noted that after the preprocessing, a target image of a set size can be obtained.
[0063] Furthermore, features of linear objects in the preprocessed target image are extracted through an adaptive curvature convolutional network to obtain image features at least in two scales.
[0064] Step 102: performing feature fusion on the image features at the at least two scales to obtain a multi-scale feature map.
[0065] In the embodiment of the present invention, the multi-scale feature map can capture the detail information of the linear object at different scales.
[0066] Specifically, feature fusion is performed on the image features of the at least two scales through at least one of upsampling, feature concatenation and addition operations to obtain a multi-scale feature map.
[0067] Step 103: Use a candidate box to select a linear object region from the multi-scale feature map to obtain a regional feature map.
[0068] It should be noted that the target image is processed by a candidate region generation algorithm to generate multiple candidate frames, and then the linear object region is selected from the multi-scale feature map using the candidate frames including the linear object region to obtain the region feature map. The candidate region generation algorithm may be a selective search algorithm or a region proposal network algorithm.
[0069] Step 104: Obtain a segmentation prediction result of the linear object according to the regional feature map.
[0070] In one implementation mode, optionally, feature extraction of linear objects in a target image is performed through an adaptive curvature convolutional network to obtain image features at least at two scales, including:
[0071] Extract features of linear objects in the target image according to the feature pyramid network to obtain image features at least at two scales;
[0072] The feature pyramid network includes at least two adaptive curvature convolution networks, one adaptive curvature convolution network is used to obtain image features of a scale, and the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network.
[0073] It should be noted that an adaptive curvature convolution network is set for each layer of the feature pyramid network, and each layer of the feature pyramid network, that is, each adaptive curvature convolution network is used to obtain image features of a scale. The layers of the feature pyramid network are connected, and the output of the adaptive curvature convolution network set in the lower layer is the input of the adaptive curvature convolution network set in the upper layer. Since the feature pyramid network is executed from the lower layer to the upper layer, the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network. Moreover, the number of layers of the feature pyramid network, that is, the number of adaptive curvature convolution networks, corresponds to the number of scales of the output image features.
[0074] It should also be noted that the input of the adaptive curvature convolutional network set in the first layer of the feature pyramid network is the target image.
[0075] like Figure 2 The figure is a flow chart of one implementation of the linear object segmentation method according to an embodiment of the present invention. According to this implementation, the linear object segmentation method includes the following steps:
[0076] Step 201: input a target image.
[0077] Step 202: preprocess the target image to obtain a preprocessed target image.
[0078] Step 203: input the preprocessed target image into the feature pyramid network.
[0079] Step 204 , obtaining image features at three scales, namely image feature 1 , image feature 2 , and image feature 3 .
[0080] Step 205: perform feature fusion on the image features at three scales.
[0081] Step 206: Obtain a multi-scale feature map.
[0082] In this embodiment, the feature pyramid network includes three layers, wherein the first layer (the bottom layer) is used to input the preprocessed target image into the adaptive curvature convolution network to obtain image feature 1, the second layer (the middle layer) is used to input image feature 1 into the adaptive curvature convolution network to obtain image feature 2, and the third layer (the top layer) is used to input image feature 2 into the adaptive curvature convolution network to obtain image feature 3.
[0083] In one embodiment, optionally, the adaptive curvature convolution network is used to deform the sampling points of the target image according to the offset so that the shape composed of the deformed sampling points conforms to the shape of the linear object.
[0084] For a 3×3 adaptive curvature convolution kernel, each position needs to learn 9 two-dimensional offsets, namely the offsets in the x and y directions, and the sampling points of the target image are offset according to the offsets. Specifically, the following formula (1) can be used to describe the mapping relationship between the position coordinates before and after the offset:
[0085]
[0086] Where x represents the input image feature; y represents the output image feature; W represents the adaptive convolution kernel; p0 represents the position of the pixel in the image feature; Δp n Represents the learned offset; R represents the set of sampling points of the convolution kernel {(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)}.
[0087] In one embodiment, optionally, the method further comprises:
[0088] The central sampling point of the target image is used as the offset starting point, and the offset of each sampling point is obtained according to the iteration direction and the offset rule, wherein the offset rule is used to indicate that the offset difference between adjacent sampling points is less than a preset threshold.
[0089] In the embodiment of the present invention, an iteration strategy is introduced in the learning of the offset of the adaptive curvature convolution network, including an iteration direction and an offset rule, wherein the offset rule indicates that the offset difference between adjacent sampling points is less than a preset threshold, thereby ensuring that the difference between the offsets of adjacent sampling points is not too large, and maintaining the continuity of the adaptive curvature convolution sampling position. For example, in the learning process of the offset of a 1×5 adaptive curvature convolution kernel, the iteration is performed along the positive direction of the x-axis, starting from the middle position p i =(x i ,y i ) starts, p i+1 Offset(1,Δy i+1 ), p i+2 Offset(2,Δy i+1 +Δy i+2 ); the iteration is performed in the opposite direction of the x-axis, then p i-1 Offset(-1,-Δy i-1 ), p i-2 Offset(-2,-Δy i-1 -Δy i-2 ), Δy i+1 , Δy i+2 , Δy i-1 and Δy i-2 Less than the preset threshold, so that the sampling position after the shift can still maintain continuity.
[0090] For any size of adaptive curvature convolution kernel, the coordinate offset of the pixel point is calculated using the following formula (2):
[0091] Pi±c=(x,yi)±(c,∑Δy±c)(2)
[0092] It should be noted that for the eigenvalues at non-integer positions, the bilinear interpolation method can be used to calculate the pixel points at the integer positions in the known image features. Taking the target pixel point as an example, the eigenvalue of the target pixel point is obtained by the linear combination of four adjacent pixel points. Specifically, the four adjacent pixel points of the target pixel point are determined to be (x1, y1), (x2, y1), (x1, y2), (x2, y2), and the eigenvalues between y1 and y2 are first linearly interpolated in the horizontal direction to obtain two temporary eigenvalues V1 and V2, and then the eigenvalues between them are linearly interpolated in the vertical direction to obtain the eigenvalue of the target pixel point.
[0093] In one implementation manner, optionally, linear object region selection is performed on the multi-scale feature map using a candidate frame to obtain a regional feature map, including:
[0094] Selecting a linear object region in the target image to generate a candidate frame;
[0095] Aligning a candidate box in the target image with the multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map;
[0096] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a region of interest;
[0097] The region of interest is processed into a preset size to obtain a regional feature map.
[0098] In an embodiment of the present invention, a target image is processed by a candidate region generation algorithm to generate multiple candidate frames; then, an alignment algorithm is used to align the candidate frames including the linear object region with the multi-scale feature map to obtain multiple candidate frames for selecting the linear object region in the multi-scale feature map; further, multiple candidate frames in the target image are accurately aligned with multiple candidate frames in the multi-scale feature map, that is, linear object regions are selected on the multi-scale feature map to obtain the region of interest; finally, the region of interest is processed into a preset size using a pooling technique, thereby simplifying feature data and calculation while retaining key feature information. In other words, the regional feature map condenses the key information in the region of interest, and the size is consistent, which is convenient for subsequent processing.
[0099] In one implementation manner, optionally, obtaining the segmentation prediction result of the linear object according to the regional feature map includes:
[0100] Generating a segmentation mask of the linear object according to the region feature map and the segmentation mask prediction network;
[0101] Removing noise and artifacts from the segmentation mask to obtain a segmentation mask after the removal process;
[0102] The segmentation mask after the removal process is binarized to obtain the segmentation prediction result of the linear object.
[0103] It should be noted that the segmentation mask is used to indicate the probability that a pixel point belongs to a linear object.
[0104] The segmentation mask prediction network includes multiple layers of convolution and deconvolution operations. The regional feature map is input into the segmentation mask prediction network, and features are extracted layer by layer and upsampled to the original size to generate a segmentation mask. Then, the noise and artifacts in the segmentation mask are removed through post-processing operations to obtain a segmentation mask after removal, thereby enhancing the details of the segmentation result. The post-processing operation can be a conditional random field or a bilateral filter. Finally, the segmentation mask after removal is binarized using a threshold segmentation method, and the portion of the segmentation mask above the threshold is set to 1, and the portion below the threshold is set to 0, thereby obtaining a segmentation prediction result of a linear object. Optionally, the threshold is set to 0.5.
[0105] Optionally, the segmentation prediction result includes at least one of the following:
[0106] the name of the linear object;
[0107] Bounding box of linear object area;
[0108] The pixel coordinates of the linear object.
[0109] like Figure 3 This is a flow chart of another implementation of the linear object segmentation method according to an embodiment of the present invention. According to this implementation, the linear object segmentation method includes the following steps:
[0110] Step 301, aligning a candidate box in a target image with a multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map.
[0111] Step 302: Use the candidate box to select the linear object region of the multi-scale feature map to obtain the region of interest.
[0112] Step 303: Using pooling technology, the region of interest is processed into a preset size to obtain a regional feature map.
[0113] Step 304: input the regional feature map into the segmentation mask prediction network to predict the segmentation result.
[0114] Step 305: Obtain segmentation prediction results.
[0115] In one embodiment, optionally, the method further comprises:
[0116] Calculate the loss function according to the segmentation prediction result and the segmentation annotation result;
[0117] Back-propagating and updating the network parameters of the segmentation mask prediction network according to the loss function;
[0118] The loss function includes at least one of the following:
[0119] Loss function corresponding to the classification task;
[0120] Loss function for bounding box selection task;
[0121] Loss function corresponding to the segmentation task.
[0122] In an embodiment of the present invention, a loss function is calculated based on the segmentation prediction result (i.e., the predicted segmentation result) and the segmentation annotation result (i.e., the annotated segmentation result), thereby guiding the back propagation and parameter update of the network parameters of the segmentation mask prediction network.
[0123] Specifically, in the classification task, the corresponding loss function is the cross entropy loss function. Specifically, using the cross entropy loss function for supervision can measure the gap between the predicted category probability distribution and the true distribution.
[0124] In the bounding box selection task, the corresponding loss function is the Smooth L1 loss function. Specifically, since the segmentation prediction results may deviate from the segmentation annotation results, the L2 (mean square error) loss function cannot be used alone, which is easy to produce too large gradients for the deviated segmentation prediction results, resulting in unstable training; while the L1 (mean absolute error) loss function processes all errors linearly. Although it is more robust to large errors, the gradients for small errors are not smooth enough, affecting the fine adjustment of the optimization process. Therefore, the Smooth L1 loss function is used, which combines the advantages of the L1 loss function and the L2 loss function. It can provide more accurate gradient information when the error is small, and maintain robustness when the error is large to prevent gradient explosion.
[0125] Optionally, the loss function corresponding to the segmentation task includes a continuous homology loss function, and the continuous homology loss function is used to optimize the segmentation prediction results of the regions with breaks and wrong connections.
[0126] In the segmentation task, the corresponding loss functions include the cross entropy loss function and the continuous homology loss function. Specifically, the cross entropy loss function is first used for supervision to achieve a higher segmentation accuracy, and then the continuous homology loss function is used to optimize the segmentation prediction results of the broken and incorrectly connected areas. The optimization process can be as follows:
[0127] First, a filter sequence is constructed, and each pixel is sorted according to the grayscale value or probability value, and the threshold is gradually increased from low to high. As the threshold increases, adjacent pixels are connected to form a one-dimensional simple structure (line segment), and then gradually form a two-dimensional simple structure (triangle) and a higher-dimensional simple structure. Each time the threshold is increased, the changes in the topological structures of different dimensions in the image are recorded, that is, the appearance and disappearance time of the topological structure in the filter sequence. These pixels are plotted on a two-dimensional plane, with the horizontal axis representing the appearance time and the vertical axis representing the disappearance time. Each pixel represents a topological structure, thus forming a continuous homology graph. In this way, the continuous homology graphs of the segmentation prediction results and the segmentation annotation results are obtained respectively, and then the similarity between the two continuous homology graphs is calculated to form a continuous homology loss function, so as to achieve the effect of further supervising the segmentation task.
[0128] Furthermore, according to the loss functions of the three tasks, the back-propagation algorithm is used to calculate the gradient of the loss relative to the network parameters, and the gradient descent algorithm (for example, Adam optimizer) is used to update the network parameters, so that the loss function is gradually reduced, which plays a role in optimizing the network parameters, thereby optimizing the segmentation prediction results and improving the classification, detection and segmentation performance.
[0129] It should be noted that the test set is used to verify the segmentation accuracy of the linear object segmentation method described in the embodiment of the present invention, and the segmentation accuracy can reach 99.09%, the average intersection-over-union ratio can reach 79.58%, and the Dice coefficient is 88.52%; the segmentation continuity is: the topological Betti numbers are 0.043 and 0.828, and the Hausdorff distance is 5.93. It can be clearly seen that the linear object segmentation method described in the embodiment of the present invention improves the segmentation accuracy and continuity.
[0130] like Figure 4 This is a flow chart of another implementation of the linear object segmentation method according to an embodiment of the present invention. According to this implementation, the linear object segmentation method includes the following steps:
[0131] Step 401, calculating the loss function according to the segmentation prediction result and the segmentation labeling result.
[0132] Step 402, respectively calculate the loss functions corresponding to the classification task, the bounding box selection task, and the segmentation task.
[0133] Step 403: In the segmentation task, the corresponding loss functions include a cross entropy loss function and a continuous homology loss function.
[0134] Step 404 , back propagation and parameter update of the network parameters of the segmentation mask prediction network are performed according to the calculated loss function.
[0135] In summary, the linear object segmentation method described in the embodiment of the present invention uses an adaptive curvature convolutional network that can dynamically adapt to complex and changeable linear object structures and extract key image features, thereby improving the efficiency and accuracy of image feature extraction, thereby improving the accuracy of the segmentation results; and a continuous homology loss function is used in the segmentation task to constrain the topological continuity of the segmentation of linear objects with complex topological structures, and optimize the segmentation prediction results of areas with breaks and incorrect connections, thereby improving the topological correctness of the segmentation results; in this way, the problem of difficult and incorrect segmentation of linear objects is solved.
[0136] like Figure 5 As shown, an embodiment of the present invention further provides a linear object segmentation device, comprising:
[0137] An extraction module 501 is used to extract features of linear objects in a target image through an adaptive curvature convolutional network to obtain image features at least at two scales;
[0138] A fusion module 502 performs feature fusion on the image features of the at least two scales to obtain a multi-scale feature map;
[0139] A selection module 503 is used to select a linear object region from the multi-scale feature map using a candidate frame to obtain a regional feature map;
[0140] The prediction module 504 is used to obtain the segmentation prediction result of the linear object according to the regional feature map.
[0141] Optionally, in the linear object segmentation device, the extraction module 501 is specifically used for:
[0142] Extract features of linear objects in the target image according to the feature pyramid network to obtain image features at least at two scales;
[0143] The feature pyramid network includes at least two adaptive curvature convolution networks, one adaptive curvature convolution network is used to obtain image features of a scale, and the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network.
[0144] Optionally, in the linear object segmentation device, the adaptive curvature convolution network is used to deform the sampling points of the target image according to the offset so that the shape composed of the deformed sampling points conforms to the shape of the linear object.
[0145] Optionally, the linear object segmentation device further comprises:
[0146] The central sampling point of the target image is used as the offset starting point, and the offset of each sampling point is obtained according to the iteration direction and the offset rule, wherein the offset rule is used to indicate that the offset difference between adjacent sampling points is less than a preset threshold.
[0147] Optionally, in the linear object segmentation device, the selection module 503 is specifically used for:
[0148] Selecting a linear object region in the target image to generate a candidate frame;
[0149] Aligning a candidate box in the target image with the multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map;
[0150] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a region of interest;
[0151] The region of interest is processed into a preset size to obtain a regional feature map.
[0152] Optionally, in the linear object segmentation device, the prediction module 504 is specifically used for:
[0153] Generating a segmentation mask of the linear object according to the region feature map and the segmentation mask prediction network;
[0154] Removing noise and artifacts from the segmentation mask to obtain a segmentation mask after the removal process;
[0155] The segmentation mask after the removal process is binarized to obtain the segmentation prediction result of the linear object.
[0156] Optionally, in the linear object segmentation device, the segmentation prediction result includes at least one of the following:
[0157] the name of the linear object;
[0158] Bounding box of linear object area;
[0159] The pixel coordinates of the linear object.
[0160] Optionally, the linear object segmentation device further comprises:
[0161] A calculation module, used for calculating a loss function according to the segmentation prediction result and the segmentation annotation result;
[0162] An updating module, configured to perform back propagation and parameter update on network parameters of the segmentation mask prediction network according to the loss function;
[0163] The loss function includes at least one of the following:
[0164] Loss function corresponding to the classification task;
[0165] Loss function for the bounding box selection task;
[0166] Loss function corresponding to the segmentation task.
[0167] Optionally, in the linear object segmentation device, the loss function corresponding to the segmentation task includes a continuous homology loss function, and the continuous homology loss function is used to optimize the segmentation prediction results of the areas with breaks and wrong connections.
[0168] The linear object segmentation device provided in the embodiment of the present invention can execute the above-mentioned linear object segmentation method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.
[0169] like Figure 6 As shown, the linear object segmentation device of an embodiment of the present invention includes: a processor 601; and a memory 602 connected to the processor 601 through a bus interface, the memory 602 is used to store programs and data used by the processor 601 when performing operations, and the processor 601 calls and executes the programs and data stored in the memory 602.
[0170] The processor 601 is used to read the program in the memory 602 and execute the following process:
[0171] The linear objects in the target image are extracted by using an adaptive curvature convolutional network to obtain image features at least at two scales.
[0172] Performing feature fusion on the image features at the at least two scales to obtain a multi-scale feature map;
[0173] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a regional feature map;
[0174] According to the regional feature map, a segmentation prediction result of the linear object is obtained.
[0175] The transceiver 603 is used to receive and send data in the controller of the processor 601 .
[0176] Among them, Figure 6In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 601 and various circuits of memory represented by memory 602 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 603 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 604 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.
[0177] The processor 601 is responsible for managing the bus architecture and general processing, and the memory 602 can store data used by the processor 601 when performing operations.
[0178] Optionally, the processor 601 is specifically configured to read the computer program and execute the following steps:
[0179] Extract features of linear objects in the target image according to the feature pyramid network to obtain image features at least at two scales;
[0180] The feature pyramid network includes at least two adaptive curvature convolution networks, one adaptive curvature convolution network is used to obtain image features of a scale, and the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network.
[0181] Optionally, the adaptive curvature convolution network is used to deform the sampling points of the target image according to the offset so that a shape composed of the deformed sampling points conforms to the shape of the linear object.
[0182] Optionally, the processor 601 is further configured to read the computer program and execute the following steps:
[0183] The central sampling point of the target image is used as the offset starting point, and the offset of each sampling point is obtained according to the iteration direction and the offset rule, wherein the offset rule is used to indicate that the offset difference between adjacent sampling points is less than a preset threshold. Optionally, the processor 601 is specifically used to read the computer program and execute the following steps:
[0184] Selecting a linear object region in the target image to generate a candidate frame;
[0185] Aligning a candidate box in the target image with the multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map;
[0186] Using a candidate box to select a linear object region from the multi-scale feature map to obtain a region of interest;
[0187] The region of interest is processed into a preset size to obtain a regional feature map.
[0188] Optionally, the processor 601 is specifically configured to read the computer program and execute the following steps:
[0189] Generating a segmentation mask of the linear object according to the region feature map and the segmentation mask prediction network;
[0190] Removing noise and artifacts from the segmentation mask to obtain a segmentation mask after the removal process;
[0191] The segmentation mask after the removal process is binarized to obtain the segmentation prediction result of the linear object.
[0192] Optionally, the segmentation prediction result includes at least one of the following:
[0193] the name of the linear object;
[0194] Bounding box of linear object area;
[0195] The pixel coordinates of the linear object.
[0196] Optionally, the processor 601 is further configured to read the computer program and execute the following steps:
[0197] Calculate the loss function according to the segmentation prediction result and the segmentation annotation result;
[0198] Back-propagating and updating the network parameters of the segmentation mask prediction network according to the loss function;
[0199] The loss function includes at least one of the following:
[0200] Loss function corresponding to the classification task;
[0201] Loss function for bounding box selection task;
[0202] Loss function corresponding to the segmentation task.
[0203] Optionally, the loss function corresponding to the segmentation task includes a continuous homology loss function, and the continuous homology loss function is used to optimize the segmentation prediction results of the regions with breaks and wrong connections.
[0204] A specific embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned linear object segmentation method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0205] In addition, an embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0206] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0207] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0208] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0209] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A linear object segmentation method, characterized in that: include: The linear objects in the target image are extracted by using an adaptive curvature convolutional network to obtain image features at least at two scales. Performing feature fusion on the image features at the at least two scales to obtain a multi-scale feature map; Using a candidate box to select a linear object region from the multi-scale feature map to obtain a regional feature map; According to the regional feature map, a segmentation prediction result of the linear object is obtained.
2. The linear object segmentation method according to claim 1, characterized in that: The linear objects in the target image are extracted through the adaptive curvature convolutional network to obtain image features at least at two scales, including: Extract features of linear objects in the target image according to the feature pyramid network to obtain image features at least at two scales; The feature pyramid network includes at least two adaptive curvature convolution networks, one adaptive curvature convolution network is used to obtain image features of a scale, and the output of the previous adaptive curvature convolution network is the input of the next adaptive curvature convolution network.
3. The linear object segmentation method according to claim 2, characterized in that: The adaptive curvature convolution network is used to deform the sampling points of the target image according to the offset so that the shape composed of the deformed sampling points conforms to the shape of the linear object.
4. The linear object segmentation method according to claim 3, characterized in that: The method further comprises: The central sampling point of the target image is used as the offset starting point, and the offset of each sampling point is obtained according to the iteration direction and the offset rule, wherein the offset rule is used to indicate that the offset difference between adjacent sampling points is less than a preset threshold.
5. The linear object segmentation method according to claim 1, characterized in that: The linear object region is selected from the multi-scale feature map using a candidate frame to obtain a regional feature map, including: Selecting a linear object region in the target image to generate a candidate frame; Aligning a candidate box in the target image with the multi-scale feature map to obtain a candidate box for selecting a linear object region in the multi-scale feature map; Using a candidate box to select a linear object region from the multi-scale feature map to obtain a region of interest; The region of interest is processed into a preset size to obtain a regional feature map.
6. The linear object segmentation method according to claim 1, characterized in that: Obtaining a segmentation prediction result of the linear object according to the regional feature map, including: Generating a segmentation mask of the linear object according to the region feature map and the segmentation mask prediction network; Removing noise and artifacts from the segmentation mask to obtain a segmentation mask after the removal process; The segmentation mask after the removal process is binarized to obtain the segmentation prediction result of the linear object.
7. The linear object segmentation method according to claim 1 or 6, characterized in that: The segmentation prediction result includes at least one of the following: the name of the linear object; Bounding box of linear object area; The pixel coordinates of the linear object.
8. The linear object segmentation method according to claim 6, characterized in that: Also includes: Calculate the loss function according to the segmentation prediction result and the segmentation annotation result; Back-propagating and updating the network parameters of the segmentation mask prediction network according to the loss function; The loss function includes at least one of the following: Loss function corresponding to the classification task; Loss function for bounding box selection task; Loss function corresponding to the segmentation task.
9. The linear object segmentation method according to claim 8, characterized in that: The loss function corresponding to the segmentation task includes a continuous homology loss function, and the continuous homology loss function is used to optimize the segmentation prediction results of the areas with breaks and wrong connections.
10. A linear object segmentation device, characterized in that: include: An extraction module, used for extracting features of linear objects in a target image through an adaptive curvature convolutional network to obtain image features at least at two scales; A fusion module, performing feature fusion on the image features of the at least two scales to obtain a multi-scale feature map; A selection module, used to select a linear object region from the multi-scale feature map using a candidate frame to obtain a regional feature map; A prediction module is used to obtain a segmentation prediction result of the linear object according to the regional feature map.
11. A linear object segmentation device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the linear object segmentation method according to any one of claims 1 to 9 is implemented.
12. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the linear object segmentation method according to any one of claims 1 to 9 is implemented.
13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the linear object segmentation method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Scene semantic segmentation method based on deep learning
CN112381097A
High-precision and light-weight height limiting rod detection algorithm
CN115171076A
Target detection method and device based on weighted deformable convolution
CN115620141A
Irregular strip-shaped object detection and reconstruction method based on image
CN116363360A
Method and device for extracting image of to-be-restored area of potential cultural relic
CN116612017A
Cited By
Image recognition system and method based on big data technology
CN119693801A
Electrical component identification method and device, storage medium and electronic equipment
CN120689898A
