Image segmentation method of SAM model in construction site scene and related device

Abstract

The invention relates to the technical field of image processing and data processing, and discloses an image segmentation method of an SAM model in a construction site scene and a related device, and the method comprises the steps: obtaining an initial construction site image data set and a to-be-segmented image data set; annotating each initial construction site image in the initial construction site image data set on a semantic mask level to obtain a training image data set; based on a segmentation and segmentation model SAM, constructing an initial hybrid convolution segmentation network; adopting the training image data set to train the initial hybrid convolution integral segmentation network to obtain a target hybrid convolution integral segmentation network; and segmenting each to-be-segmented image in the to-be-segmented image data set by adopting the target hybrid convolution segmentation network to obtain a target segmentation result. By implementing the method provided by the invention, a more accurate target segmentation result can be obtained, and the accuracy of image segmentation in a construction site scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and data processing, and in particular to an image segmentation method and related devices of a SAM model in a construction site scene. Background Art

[0002] In recent years, visual basic models have made significant progress in the field of image segmentation. Among them, the Segment Anything Model (SAM) has shown great potential in the field of intelligent construction site image segmentation based on its ability to learn complex image features. However, in the application of intelligent construction site images, it faces challenges such as low contrast, blurred boundaries, complex shapes, and small object sizes, which may lead to degraded model performance and limited generalization capabilities. For example, SAM may have difficulty distinguishing between bare soil and covered obstacles on the roadside because the foreground and background of these areas may be highly similar in shape and color. Therefore, how to achieve more accurate image segmentation in construction site scenes has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides an image segmentation method and related devices of a SAM model in a construction site scene to solve the technical problem of how to achieve more accurate image segmentation in a construction site scene, and can improve the accuracy of image segmentation in a construction site scene.

[0004] In a first aspect, a SAM model image segmentation method in a construction site scene is provided, comprising:

[0005] Obtaining an initial construction site image dataset and a to-be-segmented image dataset;

[0006] annotating each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset;

[0007] Based on the segmentation everything model SAM, an initial hybrid convolutional segmentation network is constructed;

[0008] Using the training image data set, training the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network;

[0009] The target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to obtain a target segmentation result.

[0010] In a second aspect, a SAM model image segmentation device in a construction site scene is provided, comprising:

[0011] An acquisition module, used to acquire an initial construction site image dataset and a to-be-segmented image dataset;

[0012] A first processing module is used to annotate each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset;

[0013] The second processing module is used to construct an initial hybrid convolutional segmentation network based on the segmentation everything model SAM;

[0014] A third processing module is used to train the initial hybrid convolutional segmentation network using the training image data set to obtain a target hybrid convolutional segmentation network;

[0015] The segmentation module is used to segment each image to be segmented in the image data set to obtain a target segmentation result by using the target hybrid convolutional segmentation network.

[0016] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the image segmentation method of the above-mentioned SAM model in a construction site scene are implemented.

[0017] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the image segmentation method of the above-mentioned SAM model in the construction site scene are implemented.

[0018] In the scheme implemented by the image segmentation method and related devices of the above-mentioned SAM model in the construction site scene, an initial construction site image data set and a data set of images to be segmented are obtained to annotate each initial construction site image in the initial construction site image data set at the semantic mask level to obtain a training image data set, and based on the segmentation model SAM, an initial hybrid convolutional segmentation network is constructed, so that the training image data set can be used to train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network, and then the target hybrid convolutional segmentation network is used to segment each image to be segmented in the data set of images to be segmented, so that a more accurate target segmentation result can be obtained, which can effectively improve the accuracy of image segmentation in the construction site scene, and is conducive to achieving the purpose of more accurate and efficient image segmentation in the construction site scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 It is a flow chart of an image segmentation method of a SAM model in a construction site scene in one embodiment of the present invention;

[0021] Figure 2A It is a schematic diagram of the framework structure of a target hybrid convolutional segmentation network of an image segmentation method of a SAM model in a construction site scene in one embodiment of the present invention;

[0022] Figure 2B It is a schematic diagram of the framework structure of an Attention_acmix module of an image segmentation method of a SAM model in a construction site scene in one embodiment of the present invention;

[0023] Figure 3 It is a structural schematic diagram of an image segmentation device of a SAM model in a construction site scene in one embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0025] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Since construction site images often have low contrast, fuzzy boundaries, complex shapes and small-sized objects, the SAM model has poor segmentation effect in these scenes, and its performance and generalization ability are limited. In order to solve this problem, the present invention proposes an initial mixed convolutional segmentation network (MixConvSAM, also referred to as an initial mixed convolutional segmentation model), which effectively supplements the local information in the Vision Transformer (ViT) image encoder of SAM by introducing a mixed convolution module (MixConvolution), combining convolution positioning and a window-based self-attention mechanism. At the same time, MixConvSAM reduces the deployment cost by shortening the sequence length of the ViT branch. The local features generated by the model are added by the output of the convolution path and the self-attention path, and injected into the ViT encoder, and finally the predicted image is semantically segmented.

[0028] See also Figure 1 As shown, Figure 1 A flow chart of an image segmentation method using a SAM model in a construction site scenario provided by an embodiment of the present invention includes the following steps:

[0029] S10: Obtain an initial construction site image dataset and a to-be-segmented image dataset.

[0030] The initial construction site image dataset may include one or more initial construction site images, which may be images including target category elements in a construction site scene. The target category elements may include but are not limited to bare soil, guardrails, stairs and other elements. The to-be-segmented image dataset may include one or more to-be-segmented images, which may be images that require image segmentation processing.

[0031] S20: Annotating each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset.

[0032] Annotation at the semantic mask level can be understood as the process of preprocessing the input image (i.e. the initial construction site image mentioned above) to obtain the training image dataset. By annotating each input image with different categories, such as bare soil, fences, or stairs, the subsequent model can recognize and distinguish these objects during training.

[0033] It is understandable that the training images included in the training image dataset may be images of different construction sites, different environmental conditions, and different angles. The training image dataset can provide the model with multi-dimensional, multi-scale, and multi-angle views of the target object, and the present application does not impose any restrictions on this.

[0034] Optionally, a data set containing 6500 construction site images and their corresponding label images can be obtained, wherein the image resolution can be 512×512, and the image and the corresponding label image can both be RGB images. Eighty percent of the images and their corresponding label images are randomly selected from the data set using a random algorithm (Random) to form a training set, and the remaining twenty percent can be used as a test set and a validation set, which is not limited in this application.

[0035] S30: Build an initial hybrid convolutional segmentation network based on the segmentation everything model SAM.

[0036] The initial hybrid convolutional segmentation network can be constructed based on SAM, and the initial hybrid convolutional segmentation network may include but is not limited to an image encoder, a prompt encoder, a mask decoder, and a feature fusion module added in this application, such as MixConvolution and an adapter module. The initial hybrid convolutional segmentation network can also be called a hybrid convolutional segmentation model (i.e., MixConvSAM), which can be understood as a segmentation model suitable for construction sites.

[0037] The feature fusion module MixConvolution can be located in the image encoder and can be parallel to the ViT branch in the model to provide complementary local information to the latter. And at the end, the outputs of the two branches are merged together as the final image feature embedding of MixConvSAM.

[0038] The feature fusion module can be used to receive the mask features from the mask decoder, the edge features of the edge attention module, and the original output features of the image encoder. The mask encoder can receive the image embedding from the image encoder and the position encoding information of the prompt encoder, and obtain the final output target mask information after passing through the multi-layer perceptron.

[0039] The initial hybrid convolutional segmentation network can introduce convolution positioning and window-based self-attention mechanism based on MixConvolution to effectively supplement the local information in the image encoder of SAM. Among them, convolution positioning can be performed through a position adapter (position-adapter, P-adapter), and a window-based self-attention mechanism can be implemented through a feature adapter (feature-adapter, F-adapter) to perform kernel weight projection from different positions based on input features.

[0040] S40: Using the training image dataset, train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network.

[0041] During training, MixConvSAM can use pre-trained weights to initialize its parameters, and the rest of the parameters are randomly initialized. It is understandable that only the parameters of MixConvSAM can be updated, and the other parameters remain frozen. MixConvSAM is trained using the AdamW optimizer, the learning strategy is set to warmup, the initial learning rate can be set to 0.0001, the batch size can be set to 8, and the number of iterations (epoch) can be set to 200. In the encoder part, the size of the random path dropout can be set to 0.3; in the decoder part, the size of the random inactivation (dropout) can be set to 0.1.

[0042] The model has two types of annotation supervision: object mask and object edge. For mask supervision, the feature map can be randomly sampled using an uncertainty function. Then, a function can be used to measure the overlap between the sampled area and the true area, which can effectively solve the category imbalance problem and enhance robustness.

[0043] At the same time, the similarity coefficient loss function (DiceLoss) can be used in the training process to evaluate the loss function of the model in the segmentation task. DiceLoss is derived from the similarity coefficient (Dice Coefficient), which can be used to measure the similarity between two samples. Dice Loss optimizes the model by maximizing the similarity between the predicted and true labels to encourage the model to obtain a higher overlap between the predicted and actual labels, thereby improving the segmentation accuracy of the model. Optionally, the process of calculating the classification loss through DiceLoss can be seen in the following formula:

[0044]

[0045] Among them, Dice can be used to represent the similarity coefficient, which can be used to measure the similarity between two samples. The value range of the Dice coefficient is between 0 and 1. The larger the value, the more similar the two sets are; |X∩Y| can be used to represent the number of elements in the intersection of X and Y; |X| can be used to represent the number of elements in set X, that is, the number of elements in the predicted result set; |Y| can be used to represent the number of elements in set Y, that is, the number of elements in the true label set; DiceLoss can be used to represent the similarity coefficient loss function, that is, the loss function based on the Dice coefficient, whose purpose is to minimize the Dice coefficient, thereby improving the similarity between the segmentation result and the true label.

[0046] S50: Using the target hybrid convolutional segmentation network, segment each image to be segmented in the image data set to obtain a target segmentation result.

[0047] After obtaining the target hybrid convolutional segmentation network, the target hybrid convolutional segmentation network can be used to segment the image to be segmented to obtain a more accurate target segmentation result. The process of using the target hybrid convolutional segmentation network to segment each image to be segmented in the image data set to be segmented to obtain the target segmentation result can be seen in the detailed description of the following embodiment, and this application will not repeat it here.

[0048] In this embodiment, an initial construction site image dataset and a dataset of images to be segmented are obtained, so that each initial construction site image in the initial construction site image dataset is annotated at the semantic mask level to obtain a training image dataset, and an initial hybrid convolutional segmentation network is constructed based on the segmentation everything model SAM. Thus, the training image dataset can be used to train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network, and then the target hybrid convolutional segmentation network is used to segment each image to be segmented in the dataset of images to be segmented, so that a more accurate target segmentation result can be obtained, which can effectively improve the accuracy of image segmentation in construction site scenarios, and is conducive to achieving the purpose of more accurate and efficient image segmentation in construction site scenarios.

[0049] In a feasible implementation, the target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to be segmented to obtain a target segmentation result, which refers to the process of how to apply the target hybrid convolutional segmentation network. Among them, the target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to be segmented to obtain a target segmentation result, which may include the following steps:

[0050] A1: performing position embedding processing on each image to be segmented in the image data set to obtain a position embedded feature image data set;

[0051] A2: extracting attention features from each position-embedded feature image in the position-embedded feature image dataset using a spatiotemporal attention mechanism to obtain an attention feature image dataset;

[0052] A3: extracting local features from each attention feature image in the attention feature image dataset to obtain a local feature image dataset;

[0053] A4: Decode each local feature image in the local feature image data set to obtain a target segmentation result.

[0054] See also Figure 2A , Figure 2AThe schematic diagram of the framework structure of the target mixed convolution segmentation network is shown exemplarily. Among them, the MixConvolution module in the target mixed convolution segmentation network can be composed of a block partition (Patch Partition) module and four stage (STAGE) modules connected in series in sequence. The first STAGE module is composed of a linear embedding (Linear Embedding) unit and a spatial-temporal attention mechanism block (Spatial-Temporal Attention Mechanism Block, ST-Block) connected in series. The structures of the second to fourth STAGE modules are the same, and their structures can be: patch connection (PatchMerging) module, ST-Block.

[0055] The Patch Partition module can divide the input construction site image (i.e., the above-mentioned image to be segmented) into four equal areas. The structure of the Patch Partition module can be: row and column selection layer, fully connected layer. The row and column selection layer selects two elements at intervals on the rows and columns of the feature vector output by the previous STAGE module and splices them together to form the entire tensor. The convolution kernel size of the fully connected layer can be set to 1×1. The Patch Partition module can divide the input image into four equal areas, such as image patches.

[0056] The Linear Embedding unit can flatten the four equal areas divided by the Patch Partition module. Using the linear layer, each patch can be mapped to a high-dimensional space to obtain the image block embedding features (patch embeddings), and a position code is added to each patch to preserve the position information. Patch Merging selects every 2 elements for splicing to reduce the size of the feature map, and finally forms a tensor with 4 times the number of channels. After that, the number of channels can be adjusted through the fully connected layer to return to twice the original number.

[0057] Optionally, an image encoder module consisting of F-adaper, P-adaper and MixConvolution modules can be constructed. After the input image enters the network, the input image can be first convolved to extract preliminary features, and then the feature map after the convolution operation can be processed by F-adaper. For example, the feature map can be divided into 16×16 image patches with a step size of 16 to reduce the size of the feature map by 16 times, and the number of channels can be mapped from 3 to 768 to add position encoding to the feature map after patch embedding. For example, a learnable parameter matrix can be used to add to the feature map. Deep features can be extracted through a multi-layer transformer block. It can be understood that the input image can be processed by the Patch Embedding module, F-adaper and P-adaper to perform position embedding on the image, which can make the model more sensitive to the spatial position information in the image.

[0058] For example, the image can be first input into the Patch Partition module for block division, that is, each 4x4 adjacent pixels is a patch, and then flattened in the direction of channel c. If the input is an RGB three-channel image, each patch has 16 pixels, and each pixel has three values ​​of R, G, and B, so it is 48 after flattening. In other words, after Patch Partition, the image shape can be changed from [H, W, 3] to [H / 4, W / 4, 48]. Secondly, the channel data of each pixel can be linearly transformed through the Linear Embedding layer, such as from 48 to C, that is, the image shape can be changed from [H / 4, W / 4, 48] to [H / 4, W / 4, C].

[0059] Feature maps of different sizes can be constructed through four stages. Stage 1 can first pass through a LinearEmbeding layer, and the other three stages (i.e., Stage 2, Stage 3, and Stage 4) can first pass through a Patch Merging layer for downsampling, such as expanding the receptive field layer by layer like a convolutional neural network (CNN) to obtain global information. In order to produce a hierarchical representation, as the network deepens, the number of characters (tokens) passing through PatchMeraging may gradually decrease, and its dimension will gradually expand. Each stage can change the dimension of the tensor to form a hierarchical representation. Thus, the backbone network of the existing visual task can be replaced, and then a Layer Norm layer, a global pooling layer, and a fully connected layer can be connected to further obtain the final output, and this application does not impose any restrictions on this.

[0060] In a possible implementation, extracting attention features from each position-embedded feature image in the position-embedded feature image dataset using a spatiotemporal attention mechanism to obtain an attention feature image dataset may include the following steps:

[0061] A21: performing shape transformation on each position embedding feature image in the position embedding feature image dataset to obtain a target shape feature image dataset;

[0062] A22: performing linear transformation processing on each target shape feature image in the target shape feature image dataset to obtain a linear feature image dataset;

[0063] A23: downsampling each linear feature image in the linear feature image dataset to obtain a global feature image dataset;

[0064] A24: performing an improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset;

[0065] A25: Normalize each enhanced fused feature image in the enhanced fused feature image dataset to obtain an attention feature image dataset.

[0066] As shown in Figure 2, the structure of ST-Block can be: the first normalization layer, the first attention layer, the second normalization layer, the first multilayer perceptron (MLP) layer, the third normalization layer, the second attention layer, the fourth normalization layer, and the second MLP layer. The first to fourth normalization layers can all use layer normalization (LayerNorm). The attention operation mode of the first attention layer and the second attention layer can be set to the mixed attention mode (Attention_acmix); the first MLP and the second MLP layers can be implemented using activation functions such as Gaussian Error Linear Unit (GELU), and the convolution kernel size can be set to 1×1.

[0067] In a possible implementation, performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset may include the following steps:

[0068] A241: Perform feature mapping on each global feature image in the global feature image data set to obtain a linear projection feature image;

[0069] A242: performing shift convolution calculation on the linear projection feature image according to a preset shift direction to obtain a shift convolution feature image;

[0070] A243: Calculate the self-attention weight of the shifted convolution feature image to obtain a self-attention feature image;

[0071] A244: Merge the shifted convolution feature image and the self-attention feature image to obtain an enhanced fused feature image data set.

[0072] An Attention_acmix module may be constructed to implement a process of performing improved self-attention calculation on each global feature image in the global feature image dataset based on the Attention_acmix module to obtain an enhanced fused feature image dataset.

[0073] like Figure 2B As shown, Figure 2BA schematic diagram of the framework structure of an Attention_acmix module is shown exemplarily. It can be understood that constructing an Attention_acmix module can facilitate information exchange between different windows, thereby overcoming the limitation that information cannot be transmitted between windows. Two stages can be included in the Attention_acmix module. Stage I can first perform feature mapping through 1×1 convolution to project the input features into queries, keys, and values; Stage II can include the calculation of attention weights and aggregation of value matrices to collect local features. It can be understood that compared with stage I, the corresponding computational cost in stage II is smaller, while following the same pattern as convolution.

[0074] In stage II, a specific shift direction can be used to design kernel weights by combining features from different sources with convolution kernels, and then using shift kernels with learnable parameters to perform shift operations. Optionally, based on the Attention_acmix module, each global feature image in the global feature image dataset is subjected to improved self-attention calculation to obtain an enhanced fused feature image dataset, as shown in the following formula:

[0075]

[0076]

[0077] in, represents the linear projection feature image obtained by linearly projecting the global feature image along the kernel weight of (p, q); K represents the kernel; (p, q) represents the kernel position index; f ij Represents the value of the pixel point (i, j) of the global feature image, and ij represents the pixel point index of the global feature image; represents the shifted convolution feature image; Shift represents the operation of shifted convolution calculation; p represents the ordinate of the kernel position; q represents the abscissa of the kernel position; k represents the kernel size, g represents ij represents the self-attention feature image; F out represents the enhanced fusion feature image; a represents the coefficient corresponding to the self-attention feature image; b represents the coefficient corresponding to the shifted convolution feature image.

[0078] As shown in Figure 2, the introduced attention mixing module MixConvolution image encoder can be run in parallel with the transformation module (TransBlock) branch. After obtaining the attention feature image dataset based on the MixConvolution module, local information can be extracted from the attention feature image dataset through each block of the TransBlock branch to obtain a local feature image dataset.

[0079] Optionally, a prompt encoder can be used in the target hybrid convolutional segmentation network to process the prompt information of each image input, helping the model to perform corresponding segmentation or classification tasks based on the given prompt. The output can be further passed through a decoder, such as a mask decoder, to generate a segmentation mask map and mark the classification results of different areas in the image. Integrating the embeddings output by the mask decoder and the prompt encoder, and then decoding the final segmentation mask from the integrated embedded feature map can make full use of information from different sources, which is conducive to improving the accuracy and reliability of tasks such as image segmentation.

[0080] Exemplarily, in the process of predicting bare soil, railings and stair images through the target mixed convolutional segmentation network, the bare soil, railings and stair images to be predicted can be marked with serial numbers, and the numbered images can be input into the trained MixConvSAM network in sequence to obtain the cropped image segmentation results, so that the cropped image segmentation results can be spliced ​​in sequence according to the serial numbers to obtain a more accurate final segmentation result, which is conducive to achieving the purpose of more accurate and efficient image segmentation in construction site scenarios.

[0081] Compared with the prior art, the present invention has the following advantages: First, the present invention introduces the Attention_acmix module, which can promote information exchange between different windows when only calculating the self-attention within each window, thereby overcoming the limitation that information cannot be transmitted between windows. The module uses a specific displacement direction to design the convolution kernel weights, combines features from different sources with the convolution kernel by splicing, and then uses a displacement kernel with learnable parameters to perform a displacement operation, which can improve the capacity of the model while maintaining the original alternating operation capability. In addition, multiple convolution kernel sets are used to match the output channel dimensions of the convolution path and the self-attention path, overcoming the problem of the prior art that shallow features and deep features cannot be used simultaneously during feature fusion, strengthening the context connection, and enhancing the feature extraction capability of the model, so that the present invention improves the accuracy of semantic segmentation.

[0082] Second, the present invention constructs a training network MixConvSAM, and uses the attention mixing module MixConvolution structure as the backbone network of the encoder part. The MixConvolution module uses a shifted window strategy to process large inputs in blocks to reduce computational complexity while maintaining global visual information. It combines the advantages of the Transformer self-attention mechanism and convolutional neural network, has strong modeling capabilities and good computational efficiency, breaks through the limitation of the receptive field, and overcomes the problem of loose relationship between pixels in the prior art, so that the boundaries of the image semantic segmentation images obtained by the present invention are more accurate.

[0083] Third, the present invention introduces a feature adapter and a position adapter for fine-tuning the ViT branch of the image encoder, further optimizing the SAM segmentation problem of the construction site, overcoming the problem of the prior art losing a large number of details in the downsampling process, and significantly enhancing the foreground feature extraction capability of the present invention, thereby alleviating the loss of details in the image semantic segmentation process.

[0084] In this embodiment, each image to be segmented in the image data set to be segmented is subjected to position embedding processing to obtain a position embedded feature image data set, and each position embedded feature image in the position embedded feature image data set is subjected to attention feature extraction using a spatiotemporal attention mechanism to obtain an attention feature image data set, so that each attention feature image in the attention feature image data set can be subjected to local feature extraction to obtain a local feature image data set, and then each local feature image in the local feature image data set can be decoded, so that a more accurate target segmentation result can be obtained, which is conducive to achieving the purpose of more accurate and efficient image segmentation in construction site scenarios.

[0085] The effect of the present invention can be further illustrated by the following simulation experiment:

[0086] 1. Simulation experiment conditions: The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-12700, the main frequency is 2.10GHz, the memory is 32G, and the graphics card is GeForce RTX 2080Ti; the software platform of the simulation experiment of the present invention is: Ubuntu operating system and python 3.6; the data used in the simulation experiment of the present invention comes from a construction site image dataset with an image size of 512×512, and the dataset consists of three types of objects: bare soil, stairs, and guardrails.

[0087] 2. Simulation experiment content and result analysis: The simulation experiment of the present invention is an ablation experiment using the method of the present invention and MixConvolution with an attention module added, and the simulation experiment is performed according to the following steps. The ablation experiment method is to use only the attention module MixConvolution, only the attention module MixConvolution and F-adapter, only the attention module MixConvolution and P-adapter, the attention module MixConvolution and no Attention_acmix, and only the P-adapter and F-adapter.

[0088] Step A: Randomly shuffle the samples in the entire data set, select 80% of the samples to form a training set, 10% of the samples to form a validation set, and 10% of the samples to form a test set.

[0089] Step B: Input the training set data into the MixConvSAM network for training, and train for a total of 200 epochs.

[0090] Step C: After training, evaluate the performance of the model that achieves the best validation result on the test set.

[0091] Compared with the segmentation result of the ablation experimental method, the segmentation result of the present invention has a clearer foreground contour and a more precise contour. In order to verify that the segmentation effect of the present invention is better than that of the ablation experimental method, the overall accuracy (OA) index is calculated using the following formula to evaluate the segmentation result:

[0092]

[0093] Among them, OA can be used to indicate the overall accuracy, which can refer to the proportion of pixels predicted correctly by the model to the total pixels; tp can be used to indicate the number of pixels for which both the prediction and the actual are positive, that is, the number of correctly predicted pixels; fn can be used to indicate the number of pixels for which both the prediction and the actual are negative, that is, the number of undetected pixels; N can be used to indicate the total number of pixels.

[0094] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0095] In one embodiment, a SAM model image segmentation device in a construction site scene is provided, and the SAM model image segmentation device in a construction site scene corresponds to the SAM model image segmentation method in the construction site scene in the above embodiment. Figure 3As shown, the image segmentation device of the SAM model in the construction site scene includes an acquisition module 101, a first processing module 102, a second processing module 103, a third processing module 104 and a segmentation module 105. The detailed description of each functional module is as follows:

[0096] An acquisition module 101 is used to acquire an initial construction site image dataset and a to-be-segmented image dataset;

[0097] A first processing module 102 is used to annotate each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset;

[0098] The second processing module 103 is used to construct an initial hybrid convolutional segmentation network based on the segmentation everything model SAM;

[0099] A third processing module 104 is used to train the initial hybrid convolutional segmentation network using the training image data set to obtain a target hybrid convolutional segmentation network;

[0100] The segmentation module 105 is used to segment each image to be segmented in the image data set to obtain a target segmentation result by using the target hybrid convolutional segmentation network.

[0101] In one embodiment, the segmentation module 105 is used to segment each image to be segmented in the image data set to be segmented using the target hybrid convolutional segmentation network to obtain a target segmentation result, specifically for:

[0102] Performing position embedding processing on each image to be segmented in the image data set to obtain a position embedded feature image data set;

[0103] Using a spatiotemporal attention mechanism to extract attention features from each position-embedded feature image in the position-embedded feature image dataset, to obtain an attention feature image dataset;

[0104] Extract local features from each attention feature image in the attention feature image dataset to obtain a local feature image dataset;

[0105] Each local feature image in the local feature image data set is decoded to obtain a target segmentation result.

[0106] In one embodiment, the segmentation module 105 is used to extract attention features from each position-embedded feature image in the position-embedded feature image dataset using a spatiotemporal attention mechanism to obtain an attention feature image dataset, specifically for:

[0107] Performing shape transformation on each position embedding feature image in the position embedding feature image dataset to obtain a target shape feature image dataset;

[0108] Performing linear transformation processing on each target shape feature image in the target shape feature image dataset to obtain a linear feature image dataset;

[0109] Downsampling each linear feature image in the linear feature image dataset to obtain a global feature image dataset;

[0110] Performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset;

[0111] Each enhanced fused feature image in the enhanced fused feature image dataset is normalized to obtain an attention feature image dataset.

[0112] In one embodiment, the segmentation module 105 is used to perform improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset, specifically for:

[0113] Performing feature mapping on each global feature image in the global feature image data set to obtain a linear projection feature image;

[0114] Performing shift convolution calculation on the linear projection feature image according to a preset shift direction to obtain a shift convolution feature image;

[0115] Performing self-attention weight calculation on the shifted convolution feature image to obtain a self-attention feature image;

[0116] The shifted convolution feature image and the self-attention feature image are merged to obtain an enhanced fused feature image data set.

[0117] In one embodiment, the segmentation module 105 is used to perform improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset, specifically for:

[0118] The following formula is used to implement the process of performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset:

[0119]

[0120] in, represents the linear projection feature image obtained by linearly projecting the global feature image along the kernel weight of (p, q); K represents the kernel; (p, q) represents the kernel position index; f ij Represents the value of the pixel point (i, j) of the global feature image, and ij represents the pixel point index of the global feature image; represents the shifted convolution feature image; Shift represents the operation of shifted convolution calculation; p represents the ordinate of the kernel position; q represents the abscissa of the kernel position; k represents the kernel size, g represents ij represents the self-attention feature image; F out represents the enhanced fusion feature image; a represents the coefficient corresponding to the self-attention feature image; b represents the coefficient corresponding to the shifted convolution feature image.

[0121] The present invention provides an image segmentation device of a SAM model in a construction site scene. An initial construction site image data set and a to-be-segmented image data set are obtained to annotate each initial construction site image in the initial construction site image data set at a semantic mask level to obtain a training image data set, and an initial hybrid convolutional segmentation network is constructed based on a segmentation model SAM. Thus, the training image data set can be used to train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network, and then the target hybrid convolutional segmentation network is used to segment each to-be-segmented image in the to-be-segmented image data set, so that a more accurate target segmentation result can be obtained, which can effectively improve the accuracy of image segmentation in the construction site scene, and is conducive to achieving the purpose of more accurate and efficient image segmentation in the construction site scene.

[0122] For the specific definition of the image segmentation device of the SAM model in the construction site scene, please refer to the definition of the image segmentation method of the SAM model in the construction site scene above, which will not be repeated here. The various modules in the image segmentation device of the SAM model in the construction site scene can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps of the service side of a SAM model image segmentation method in a construction site scene are realized.

[0124] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a SAM model image segmentation method in a construction site scene are implemented.

[0125] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0126] Obtaining an initial construction site image dataset and a to-be-segmented image dataset;

[0127] annotating each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset;

[0128] Based on the segmentation everything model SAM, an initial hybrid convolutional segmentation network is constructed;

[0129] Using the training image data set, training the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network;

[0130] The target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to obtain a target segmentation result.

[0131] The present invention provides a computer device, which obtains an initial construction site image data set and a data set of images to be segmented, so as to annotate each initial construction site image in the initial construction site image data set at the semantic mask level to obtain a training image data set, and constructs an initial hybrid convolutional segmentation network based on a segmentation everything model SAM, so that the training image data set can be used to train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network, and then the target hybrid convolutional segmentation network is used to segment each image to be segmented in the data set of images to be segmented, so as to obtain a more accurate target segmentation result, effectively improve the accuracy of image segmentation in construction site scenes, and facilitate achieving the purpose of more accurate and efficient image segmentation in construction site scenes.

[0132] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0133] Obtaining an initial construction site image dataset and a to-be-segmented image dataset;

[0134] annotating each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset;

[0135] Based on the segmentation everything model SAM, an initial hybrid convolutional segmentation network is constructed;

[0136] Using the training image data set, training the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network;

[0137] The target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to obtain a target segmentation result.

[0138] The present invention provides a computer-readable storage medium, which obtains an initial construction site image data set and a data set of images to be segmented, so as to annotate each initial construction site image in the initial construction site image data set at the semantic mask level to obtain a training image data set, and constructs an initial hybrid convolutional segmentation network based on a segmentation everything model SAM, so that the training image data set can be used to train the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network, and then the target hybrid convolutional segmentation network is used to segment each image to be segmented in the data set of images to be segmented, so as to obtain a more accurate target segmentation result, effectively improve the accuracy of image segmentation in construction site scenes, and help achieve the purpose of more accurate and efficient image segmentation in construction site scenes.

[0139] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0142] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A SAM model image segmentation method in a construction site scene, characterized in that: The method comprises: Obtaining an initial construction site image dataset and a to-be-segmented image dataset; annotating each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset; Based on the segmentation everything model SAM, an initial hybrid convolutional segmentation network is constructed; Using the training image data set, training the initial hybrid convolutional segmentation network to obtain a target hybrid convolutional segmentation network; The target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to obtain a target segmentation result.

2. The image segmentation method of the SAM model in the construction site scene according to claim 1 is characterized in that: The target hybrid convolutional segmentation network is used to segment each image to be segmented in the image data set to obtain a target segmentation result, including: Performing position embedding processing on each image to be segmented in the image data set to obtain a position embedded feature image data set; Using a spatiotemporal attention mechanism to extract attention features from each position-embedded feature image in the position-embedded feature image dataset, to obtain an attention feature image dataset; Extract local features from each attention feature image in the attention feature image dataset to obtain a local feature image dataset; Each local feature image in the local feature image data set is decoded to obtain a target segmentation result.

3. The image segmentation method of the SAM model in the construction site scene according to claim 2 is characterized in that: The step of extracting attention features from each position-embedded feature image in the position-embedded feature image dataset using a spatiotemporal attention mechanism to obtain an attention feature image dataset comprises: Performing shape transformation on each position embedding feature image in the position embedding feature image dataset to obtain a target shape feature image dataset; Performing linear transformation processing on each target shape feature image in the target shape feature image dataset to obtain a linear feature image dataset; Downsampling each linear feature image in the linear feature image dataset to obtain a global feature image dataset; Performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset; Each enhanced fused feature image in the enhanced fused feature image dataset is normalized to obtain an attention feature image dataset.

4. The image segmentation method of the SAM model in a construction site scene according to claim 3 is characterized in that: The step of performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset comprises: Performing feature mapping on each global feature image in the global feature image data set to obtain a linear projection feature image; Performing shift convolution calculation on the linear projection feature image according to a preset shift direction to obtain a shift convolution feature image; Performing self-attention weight calculation on the shifted convolution feature image to obtain a self-attention feature image; The shifted convolution feature image and the self-attention feature image are merged to obtain an enhanced fused feature image data set.

5. The image segmentation method of the SAM model in a construction site scene according to claim 3 is characterized in that: The step of performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset comprises: The following formula is used to implement the process of performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset: in, represents the linear projection feature image obtained by linearly projecting the global feature image along the kernel weight of (p, q); K represents the kernel; (p, q) represents the kernel position index; f ij Represents the value of the pixel point (i, j) of the global feature image, and ij represents the pixel point index of the global feature image; represents the shifted convolution feature image; Shift represents the operation of shifted convolution calculation; p represents the ordinate of the kernel position; q represents the abscissa of the kernel position; k represents the kernel size, g represents ij represents the self-attention feature image; F out represents the enhanced fusion feature image; a represents the coefficient corresponding to the self-attention feature image; b represents the coefficient corresponding to the shifted convolution feature image.

6. An image segmentation device of a SAM model in a construction site scene, characterized in that: The image segmentation device of the SAM model in the construction site scene includes: An acquisition module, used to acquire an initial construction site image dataset and a to-be-segmented image dataset; A first processing module is used to annotate each initial construction site image in the initial construction site image dataset at a semantic mask level to obtain a training image dataset; The second processing module is used to construct an initial hybrid convolutional segmentation network based on the segmentation everything model SAM; A third processing module is used to train the initial hybrid convolutional segmentation network using the training image data set to obtain a target hybrid convolutional segmentation network; The segmentation module is used to segment each image to be segmented in the image data set to obtain a target segmentation result by using the target hybrid convolutional segmentation network.

7. The image segmentation device of the SAM model in the construction site scene according to claim 6 is characterized in that: The segmentation module is used to segment each image to be segmented in the image data set to be segmented using the target hybrid convolutional segmentation network to obtain a target segmentation result, specifically for: Performing position embedding processing on each image to be segmented in the image data set to obtain a position embedded feature image data set; Using a spatiotemporal attention mechanism to extract attention features from each position-embedded feature image in the position-embedded feature image dataset, to obtain an attention feature image dataset; Extract local features from each attention feature image in the attention feature image dataset to obtain a local feature image dataset; Each local feature image in the local feature image data set is decoded to obtain a target segmentation result.

8. The image segmentation device of the SAM model in the construction site scene according to claim 7 is characterized in that: The segmentation module is used to extract attention features from each position embedding feature image in the position embedding feature image dataset using a spatiotemporal attention mechanism to obtain an attention feature image dataset, specifically for: Performing shape transformation on each position embedding feature image in the position embedding feature image dataset to obtain a target shape feature image dataset; Performing linear transformation processing on each target shape feature image in the target shape feature image dataset to obtain a linear feature image dataset; Downsampling each linear feature image in the linear feature image dataset to obtain a global feature image dataset; Performing improved self-attention calculation on each global feature image in the global feature image dataset to obtain an enhanced fused feature image dataset; Each enhanced fused feature image in the enhanced fused feature image dataset is normalized to obtain an attention feature image dataset.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image segmentation method of the SAM model in the construction site scene is implemented as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image segmentation method of the SAM model in a construction site scene is implemented.

Images