Image segmentation and feature extraction method

Abstract

The invention provides an image segmentation and feature extraction method, which comprises the following steps of S1, segmenting an original image by using a semantic segmentation model ToolNet to obtain a mask image, and segmenting the mask image to obtain a segmented image; s2, inputting the segmented image into a feature extraction network, converting the segmented image into a feature vector with rich representation capability, and extracting feature information; according to the method, the semantic segmentation model ToolNet is adopted to segment the original image, the ToolRSU module based on octave convolution is adopted to decompose the image features according to the frequency, the space redundancy and calculation cost of the model are reduced, a double attention mechanism is introduced to improve the segmentation precision of the semantic segmentation model ToolNet, and the extraction precision and efficiency of the image features are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular, relates to an image segmentation and feature extraction method. Background Art

[0002] In the existing image segmentation and feature extraction methods, deep learning models and U-Net semantic segmentation network models are usually used to segment images to obtain feature information, and a simple framework is designed through the U-Net network model to extract feature information to achieve image recognition. The U-Net network model consists of three parts: encoding, decoding, and feature fusion. In the encoding process, multi-layer downsampling is used to extract more comprehensive features; in the decoding process, multi-layer upsampling is used to restore the feature results; and in the feature fusion stage, the feature results are generated in the fusion process to generate a significant probability map. U-Net network model or U 2 -Net network model encoding and decoding process, need to use a single convolution layer or deconvolution layer for each sample, due to the use of small convolution kernel resulting in too narrow receptive field, can only extract local details of the defect. At the same time, due to the use of dilated convolution leads to excessive calculation problems, resulting in low accuracy and efficiency of extracting image features. Summary of the invention

[0003] The purpose of the present invention is to provide an image segmentation and feature extraction method, which uses a semantic segmentation model ToolNet to segment the original image, and adopts the ToolRSU module based on octave convolution to decompose the image features by frequency, thereby reducing the spatial redundancy and computational cost of the model, and introducing a dual attention mechanism to improve the segmentation accuracy of the model, thereby improving the accuracy and efficiency of image feature extraction.

[0004] The present invention is implemented in that an image segmentation and feature extraction method comprises the following steps:

[0005] S1, use the semantic segmentation model ToolNet to segment the original image to obtain a mask map, and segment the mask map to obtain a segmented image;

[0006] S2, inputs the segmented image into the feature extraction network, converts it into a feature vector with rich representation capabilities, and extracts the feature vector.

[0007] In some feasible implementations, a dual attention mechanism is used to improve the segmentation accuracy of the semantic segmentation model ToolNet, where the dual attention mechanism is two attention mechanisms including temporal attention and spatial attention mechanisms.

[0008] In some feasible implementations, a ToolRSU module is used to improve the segmentation accuracy of the semantic segmentation model ToolNet; the ToolRSU module uses octave convolution to use feature information of local context and global context to improve the segmentation effect.

[0009] Furthermore, the semantic segmentation model ToolNet includes an encoding layer, a decoding layer and a fully connected layer, and each layer includes at least one U-shaped residual block.

[0010] Furthermore, the encoding layer has a six-layer structure, and the decoding layer has a five-layer structure.

[0011] Furthermore, the feature map X and the convolution kernel W are input into the octave convolution layer, and the input feature map X and the convolution kernel W in the convolution layer are divided into high-frequency components and low-frequency components.

[0012]

[0013] The low-frequency component refers to the feature map obtained after Gaussian filtering, and the high-frequency component refers to the original feature map without Gaussian filtering; L and X H Respectively represent the low-frequency component and high-frequency component of the feature map, W L and W H Represent the convolution kernels for low-frequency components and high-frequency components respectively.

[0014] Furthermore, while extracting the features of low-frequency components and high-frequency components in octave convolution, the feature representations of low-frequency components and high-frequency components communicate effectively with each other.

[0015] Furthermore, when the feature information is updated from low frequency to high frequency, the low frequency component needs to be upsampled, and the calculation formula is:

[0016] Y H =f(X H ; W H→H )+Upsampling(f(X L ,W L→H ),k)

[0017] Where f(X;W) represents convolution with kernel W parameters, Upsampling(f(),k) represents upsampling, and W L→H It represents the process of updating from low frequency to high frequency, k represents the stride, and the nearest neighbor interpolation with stride k=2.

[0018] Furthermore, when the feature information is updated from high frequency to low frequency, average pooling is required to downsample the high frequency components. The calculation formula is:

[0019] Y L =f(XL ; W L→L )+f(pool(X H ,k);W H→L )

[0020] Where pool(X,k) represents the average pooling operation to achieve downsampling, W H→L It represents the process of updating information from high frequency to low frequency, k represents the stride; stride k=2.

[0021] Furthermore, the feature map size of the low-frequency component is set to half of the feature map size of the high-frequency component.

[0022] Preferably, the feature extraction network uses the lightweight network EfficientNetV2-s.

[0023] Furthermore, S2 includes:

[0024] S21, input the segmented image into the feature extraction network, and perform shallow feature extraction on the segmented image after convolution with a convolution kernel to obtain feature information A1X;

[0025] S22, feature information A1X is convolved by Fused-MBConv to further perform feature extraction and information fusion to obtain feature information A2X;

[0026] S23, feature information A2X is further subjected to feature extraction and information fusion through MBConv convolution to obtain feature information A3X;

[0027] S24, feature information A3X is globally averaged pooled to reduce the spatial dimension of the final feature map to 1x1;

[0028] S25, mapping the feature information A3X to the probability distribution of the object target category through a fully connected layer to obtain an object feature vector with a dimension of 512.

[0029] Preferably, in S21, the size of the convolution kernel is 3×3.

[0030] Furthermore, each Fused-MBConv convolution block contains five steps: decomposition of the input feature map, depthwise separable convolution, pointwise convolution, fusion, and residual connection.

[0031] Furthermore, low-frequency features Upsample from a layer in the encoding stage to the same scale as the high-frequency component It is used for channel splicing, and then the feature map is converted into a spatial attention weight map through a 1*1 convolution operation; and Represent the feature maps of high-frequency components and low-frequency components of the i-th layer respectively.

[0032] Furthermore, for the high-frequency component of the feature map, the generated attention weight map is first normalized by the sigmoid function, and then directly multiplied with the feature map of the high-frequency component to obtain the high-frequency feature map

[0033]

[0034] Where δ() represents the sigmoid function, Conv 1X1 () represents the 1*1 convolution after channel splicing and fusion; It means that the generated spatial attention map is multiplied by the corresponding point of each high-frequency feature map.

[0035] Furthermore, for the low-frequency component of the feature map, the generated attention weight map is first downsampled to the same scale as the low-frequency component, then normalized using the sigmoid function, and finally multiplied with the low-frequency component to obtain the low-frequency feature map.

[0036]

[0037] Where Sub() represents the downsampling function; It means that the generated spatial attention map is multiplied by the corresponding point of each low-frequency feature map.

[0038] Preferably, the input size of the feature extraction network is 224×224×3.

[0039] Compared with the prior art, the image segmentation and feature extraction method provided by the present invention has at least the following beneficial effects:

[0040] 1. The present invention adopts the semantic segmentation model ToolNet to segment the original image, and uses the ToolRSU module based on octave convolution to decompose the image features by frequency, reducing the spatial redundancy and computational cost of the model, and introduces a dual attention mechanism to improve the segmentation accuracy of the semantic segmentation model ToolNet, thereby improving the accuracy and efficiency of image feature extraction.

[0041] 2. The present invention further adopts the method of combining Fused-MBConv convolution with MBConv convolution to further improve the segmentation accuracy of the semantic segmentation model ToolNet. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a structure diagram of the semantic segmentation model ToolNet in the present invention;

[0044] Figure 2 The octave convolution in the present invention performs feature extraction with high frequency and low frequency as shown in Figure 1;

[0045] Figure 3 This is the second diagram showing that the octave convolution in the present invention performs feature extraction with high frequency and low frequency. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. For those skilled in the art, the present invention may have various changes and variations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0047] The image segmentation and feature extraction method of the present invention is applied to the field of object recognition. In the process of object recognition, the collected object image is usually segmented to form feature information, and then the feature information is extracted. Finally, the extracted feature information is compared with the feature information of the feature library to realize the recognition of the object. A target category label set is predefined in the feature library, and the target category is used to represent different object targets to be recognized. For example, for image classification tasks, the target category can be a cat, a dog, a car, an airplane, etc. First, the image is collected by using a camera, and can be transmitted through a network streaming method to realize remote acquisition of the image, real-time transmission of the detection image, and further detection of whether there is an object target in the image, and determine its position. In this application, the semantic segmentation model ToolNet is used to segment the collected original image to obtain a mask map, and the segmentation mask map is segmented to obtain a segmented image; the foreground and background of the segmented image are binarized, and multi-scale feature information is formed; these feature information can be information such as the shape, texture, color, etc. of the object captured, or the relationship between the object and the surrounding environment. Then the segmented image is input into the feature extraction network, converted into a feature vector with rich representation capabilities, and the feature vector is extracted. For details, see Figure 1 , the semantic segmentation model ToolNet includes an encoding layer, a decoding layer and a fully connected layer; each layer contains at least one U-shaped residual block for extracting multi-scale features. The encoding layer path is usually responsible for gradually reducing the size and number of channels of the feature map to extract higher-level semantic information; while the decoding layer path restores the size of the feature map by upsampling and skip connections, retains details and low-level features, and helps to extract multi-scale features. The feature map is the direct intermediate transmission data of each layer, and the mask map is the result of image segmentation. The mask map plus the original image can obtain the segmented image. In this application, the reason why the U-shaped residual block is used to extract multi-scale features is that its structure allows information to be skipped between the encoding layer and the decoding layer path, thereby promoting the transmission and fusion of features. Under the condition of minimizing the operation as much as possible, the U-shaped residual block can be used as much as possible for multi-scale feature extraction. Furthermore, in order to improve the operation speed and segmentation accuracy, after a large number of experiments, the best effect is achieved when the encoding layer is a six-layer structure and the decoding layer is a five-layer structure.

[0048] Each time a training sample passes through an encoding layer, the sampled training sample is passed downward, and the training result of this layer is passed to the decoding layer of the same level to calculate the loss function. The calculation method of the loss function depends on the specific type of loss function used. The loss function is a function used to measure the degree of difference between the model prediction result and the true label. Common neural network loss functions include Mean Squared Error (MSE), Cross-Entropy Loss and Log Loss. The main significance is to use evaluation indicators or formula methods from disciplines such as mathematics and information theory to measure the degree of difference between the model prediction result and the true label. For example: Cross-Entropy Loss:

[0049] Cross-Entropy Loss = -Σ(true label * log(prediction probability))

[0050] For the entire training set, the cross entropy loss of each sample can be summed and then averaged:

[0051] Cross-Entropy Loss = (-1 / n)*Σ(true label*log(prediction probability))

[0052] If the training samples used are maintenance tools, first select common maintenance tool categories and build a data acquisition and detection experimental platform consisting of a black background board and a 4-megapixel high-definition camera. Through image acquisition and data enhancement, the original acquired feature image and the mask image obtained after segmentation by the semantic segmentation model ToolNet are a pair of training samples.

[0053] In some feasible implementations, the ToolRSU module can be used to improve the segmentation accuracy of the semantic segmentation model ToolNet. Figure 2 , Octave convolution is used in the ToolRSU module to use the feature information of local context and global context to improve the segmentation effect. Unlike general ordinary convolution, octave convolution considers that the feature map of the input or output of the convolution layer, or the channel has high-frequency components and low-frequency components, which can capture the overall trend and local details of the image at the same time, thereby improving the effect of object recognition. Low-frequency components are used to support the overall shape of the object, but they are often redundant, which can be alleviated in the encoding process. High-frequency components are used to restore the edges and detailed textures of the original image. Among them, local context refers to the information obtained in a relatively small area around a given pixel. It mainly focuses on the local structure and details around the pixel to capture the local features adjacent to the pixel. For example, in the image segmentation task, the pixel can be inferred to which category by observing the texture, edge and color information around the pixel. Global context refers to the global information obtained within the entire image range. It focuses on the semantics and contextual relationships of the entire image to understand the connection and association between different regions in the image. For example, in the image segmentation task, the categories of different regions can be inferred by observing the semantic content, object shape and layout of the entire image.

[0054] See also Figure 2 , Figure 3 In the octave convolution, the low-frequency component refers to the feature map obtained after Gaussian filtering, and the high-frequency component refers to the original feature map without Gaussian filtering. Due to the redundancy of the low-frequency component, the feature map size of the low-frequency component is set to half of the feature map size of the high-frequency component. The input feature map X and the convolution kernel W in the convolution layer are divided into high-frequency and low-frequency components, as shown in formula (1):

[0055]

[0056] Where X L and X H Respectively represent the low-frequency component and high-frequency component of the feature map, W L and W H Represent the convolution kernels for low-frequency components and high-frequency components respectively.

[0057] Since the sizes of high-frequency and low-frequency feature maps are inconsistent, convolution operations cannot be performed. While extracting the features of low-frequency components and high-frequency components in octave convolution, effective communication between the features of low-frequency components and high-frequency components is required. It can be understood that the input feature map is decomposed into low-frequency components and high-frequency components. The low-frequency component can then be passed to the low-frequency component processor of the next layer, while the high-frequency component is passed to the high-frequency component processor of the next layer. In the low-frequency component processor of the next layer, the low-frequency component can be further processed and feature extracted. Then, through cross-layer connections, the processed low-frequency component is fused with the high-frequency component of the next layer. This fusion can be a simple element-level addition or use other complex operations. In this way, the low-frequency component can utilize the detail information of the high-frequency component, thereby improving the representation ability of the feature. Similarly, in the high-frequency component processor of the next layer, the high-frequency component can be further processed and fused with the low-frequency component of the previous layer. In this way, the high-frequency component can utilize the overall trend information of the low-frequency component to better capture local details. Through cross-layer connections and feature fusion, effective information transmission and communication can be carried out between the low-frequency component and the high-frequency component. This communication mechanism can make low-frequency and high-frequency features complement each other, enhance the multi-scale representation ability of objects, and improve the performance of object recognition.

[0058] Specifically, in order to achieve effective communication between the features of the high-frequency component and the low-frequency component, when the information is updated from low frequency to high frequency (process W L→H ), the low-frequency components need to be upsampled, see formula (2).

[0059] Y H =f(X H ; W H→H )+Upsampling(f(X L ,w L→H ),k) (2)

[0060] Where f(X;W) represents convolution with convolution kernel W parameters, Upsampling(f(),k) represents upsampling, and the nearest neighbor interpolation calculation with stride k=2 is used.

[0061] When information is updated from high frequency to low frequency (process W H→L ), average pooling is required to downsample the high-frequency components, see formula (3).

[0062] Y L =f(X L ; W L→L )+f(pool(X H ,k);W H→L ) (3)

[0063] Where pool(X,k) represents the average pooling operation to achieve downsampling, with a stride of k = 2.

[0064] Average pooling is an operation that reduces the size of feature maps. It averages the feature values ​​in each region to obtain the downsampled feature values. The advantage of this is that it can retain certain feature information, keep the height and width dimensions of low-frequency and high-frequency feature maps consistent, and reduce the dimension of the feature map so that it can be sent to the next calculation operator for operation, effectively reducing the computational complexity, thereby updating the high-frequency and low-frequency feature map data. By using average pooling to downsample the high-frequency components, it is helpful to obtain and extract more abstract and global feature representations, and improve performance in subsequent classification, recognition and other tasks.

[0065] It should be noted that the nearest neighbor interpolation is an image interpolation method used to estimate new pixel values ​​on discrete coordinates. In nearest neighbor interpolation, the value of the new pixel is obtained by finding the nearest pixel in the original image and copying it. Specifically, for each new pixel position to be inserted, the nearest neighbor interpolation finds the nearest pixel in the original image and assigns its value to the new pixel. Nearest neighbor interpolation is a simple and fast interpolation method. By using the nearest neighbor interpolation calculation with a stride k = 2, the size of the feature map can be effectively reduced, which can reduce the amount of calculation and memory consumption, while retaining a certain degree of feature information for subsequent processing and analysis.

[0066] Please continue to see Figure 1 After the training sample traverses all encoding layers and decoding layers, 6 feature results will be extracted, and these feature results will be restored to the same size as the training sample through upsampling. The training sample can be a 4x4 image, or upsampled to an 8x8 image. The specific balance between accuracy and speed can be based on different scenarios. Different evaluations and adjustments are required. Finally, all feature results are aggregated to the fully connected layer, and the prediction results of each decoding extreme are upsampled to the same size as the original image. The decoding extreme refers to the upsampling operation on the feature map in the segmentation task, mapping the low-resolution feature map back to the size of the original image to obtain detailed and accurate segmentation results.

[0067] Please continue to see Figure 1In some feasible implementations, a dual attention mechanism is used to improve the segmentation accuracy of the semantic segmentation model ToolNet. The dual attention mechanism is a technology for feature fusion, which can effectively combine feature information at different levels or different channels to improve the performance and accuracy of the semantic segmentation model ToolNet; it enables the semantic segmentation model ToolNet to adaptively focus on important information in different feature maps by learning the attention mechanism at the weight or channel level. In this process, through cascade operations, feature maps at different levels or decoding extremes are connected or fused layer by layer to form a multi-scale feature representation. This cascade method can capture the multi-scale information of objects, making the semantic segmentation model ToolNet more robust when processing images of different scales. Through continuous iteration, the semantic segmentation model ToolNet can gradually optimize the prediction results and improve the effect of object segmentation by reducing the value of the loss function. In the iterative process, the semantic segmentation model ToolNet can learn more refined feature representations and more accurate prediction results, thereby continuously improving the performance of object recognition and improving the segmentation effect of the semantic segmentation model ToolNet.

[0068] In this application, the dual attention mechanism includes two attention mechanisms: time attention and spatial attention mechanism; among them, the time attention mechanism mainly focuses on the information in the sequence or time dimension. In the semantic segmentation task, this means that the model is able to focus on different features at different time steps, and the time attention mechanism can dynamically adjust their weights according to the importance of the features at different time steps; in this way, the model can better capture the key time-related features in the sequence, thereby improving the accuracy of segmentation. The spatial attention mechanism mainly focuses on the information in the image or spatial dimension. In the semantic segmentation task, the model needs to understand the contextual relationship between different regions in order to better segment different objects or regions in the image; the spatial attention mechanism can adjust their weights according to the importance of different regions, so that the semantic segmentation model ToolNet can better focus on important spatial locations and ignore irrelevant regions. This helps to improve the segmentation model's ability to capture target boundaries and details. By combining the time attention and spatial attention mechanisms, the dual attention mechanism can capture important time and space information more comprehensively in the semantic segmentation model. This can help improve segmentation accuracy because the semantic segmentation model ToolNet can more accurately focus on important time steps and spatial regions, thereby better distinguishing different target categories and reducing misclassification. The dual attention mechanism can better capture sequence and spatial information in the semantic segmentation task through temporal attention and spatial attention mechanism, thereby improving the accuracy and performance of the segmentation model. In some feasible implementations, the semantic segmentation model ToolNet can also be trained with a large number of object image test samples to evaluate the performance and accuracy of the semantic segmentation model ToolNet.

[0069] See also Figure 2 , Figure 3 The feature map obtained by octave convolution extraction has high-frequency components and low-frequency components. In this application, the feature information of high-frequency and low-frequency feature maps is fused through a dual attention mechanism. The spatial attention mechanism is used to enhance the important information in the feature map, and the high-frequency components and low-frequency components are processed in a dual-frequency manner at the same time.

[0070] Low frequency characteristics Upsample from a layer in the encoding stage to the same scale as the high-frequency component It is used for channel splicing, and then the feature map is converted into a spatial attention weight map through a 1*1 convolution operation.

[0071] For the high-frequency component of the feature map, the generated attention weight map is first normalized by the sigmoid function, and then directly multiplied with the feature map of the high-frequency component to obtain the high-frequency feature map See formula (4).

[0072]

[0073] Where δ() represents the sigmoid function, Conv 1X1 () represents the 1*1 convolution after channel splicing and fusion. It means that the generated spatial attention map is multiplied by the corresponding point of each high-frequency feature map. and Represent the feature maps of high-frequency components and low-frequency components of the i-th layer respectively.

[0074] For the low-frequency component of the feature map, the generated attention weight map is first downsampled to the same scale as the low-frequency component, then normalized using the sigmoid function, and finally multiplied with the low-frequency component to obtain the low-frequency feature map. See formula (5):

[0075]

[0076] Where Sub() represents the downsampling function. By adopting the above dual attention structure, effective communication between the features of low-frequency components and high-frequency components is achieved. At the same time, the high- and low-frequency feature maps of a certain layer in the encoding stage are transferred to the corresponding layer in the decoding stage for feature fusion after feature enhancement.

[0077] The original image of the collected object is segmented using the semantic segmentation model ToolNet to obtain a mask map, the mask map is segmented to obtain a segmented image, and the segmented image is input into the feature extraction network to convert it into a feature vector with rich representation capabilities. For example, the feature vectors of objects such as tape, pliers, and wrenches are predefined in the feature library. The corresponding feature vectors are obtained by inputting a single corresponding object image and adding it to the feature library. Generally, the feature vector of the mask map can be obtained in the last layer or the last few layers of the feature extraction network. These feature vectors can be regarded as high-level representations of the input feature map in the network, with more abstract and semantic features. These feature vectors can contain information about the shape, texture, color, and other related attributes of the object. The feature extraction network refers to the entire deep learning model or neural network structure. The feature extraction network includes multiple Fused-MBConv convolution blocks. In this application, the feature extraction network uses a lightweight network EfficientNetV2-s.

[0078] Specifically, S21, the segmented image is input into the feature extraction network, and the input size of the feature extraction network is 224×224×3. The size 224×224×3 means that the height of the segmented image is 224 pixels, the width is 224 pixels, and it contains 3 channels, such as RGB images. This size is a commonly used input feature map size in computer vision tasks because it has the advantages of balance, model adaptability, and object adaptability. Among them, the balance is that the size of 224×224 can maintain the balance of the segmented image in most cases, neither too small to cause information loss, nor too large to increase the computational burden. Model adaptability means that this size is used in many pre-trained computer vision models, so these pre-trained models can be directly used for transfer learning, saving training time and resource consumption. Object adaptability means that for tasks such as object recognition, the input size of 224×224 can adapt to the aspect ratio of different objects, ensuring that the segmented image will not be distorted or stretched due to different object shapes. After convolution with a convolution kernel, shallow feature extraction is performed on the segmented image to obtain feature information A1X. The value of the convolution kernel in this case is usually based on feature receptive field and computational efficiency; the feature receptive field means that a smaller convolution kernel can capture local detail features, such as edges, textures, etc.; and in the shallow feature extraction stage, it is reasonable to pay more attention to the local features of the segmented image. Computational efficiency means that a smaller convolution kernel has fewer parameters and less computation than a larger convolution kernel. Therefore, in this application, the size of the convolution kernel is 3×3, which is more suitable for shallow feature extraction tasks.

[0079] S22, feature information A1X is further subjected to Fused-MBConv convolution for feature extraction and information fusion to obtain feature information A2X. Although more Fused-MBConv convolution blocks can be used, in this application, after repeatedly adjusting the model calculation amount in the balanced deployment and the accuracy required by the task, 3 Fused-MBConv convolution blocks are selected for feature extraction and information fusion for better results. Each Fused-MBConv convolution block includes five steps: decomposition of the input feature map, depth-separable convolution, point-by-point convolution, fusion, and residual connection, where:

[0080] Decomposition of input feature map: The input feature map is decomposed by channels, dividing the channel information into different sub-channel groups, so that the input feature map can be divided into multiple parts, thereby facilitating finer-grained feature extraction; that is, the segmented image is divided into multiple parts, thereby facilitating finer-grained feature extraction.

[0081] Depthwise separable convolution is a convolution operation consisting of two steps: depthwise convolution and pointwise convolution. It has fewer parameters and computational complexity while maintaining effective feature extraction. By reducing parameters and computation, it can improve the computational efficiency of the model and reduce the storage requirements of the model while maintaining accuracy.

[0082] Deep convolution refers to the convolution operation on the channel dimension, which uses a small convolution kernel, usually 3x3 or 5x5, to perform convolution on each channel of the input. The purpose of deep convolution is to capture local features such as edges, textures, etc. independently for each channel. By performing convolution operations on each channel, feature representations within the channel can be learned.

[0083] Point-wise convolution, also known as 1x1 convolution, is the application of a 1x1 convolution kernel on each channel. It achieves information interaction and integration between channels by performing convolution operations on each channel. Point-wise convolution can be viewed as a linear combination and adjustment of features in the channel dimension to obtain a new feature representation. It can be used to reduce the number of channels, adjust the relationship between channels of feature maps, or introduce nonlinear transformations.

[0084] Depthwise separable convolution combines depthwise convolution and pointwise convolution together. It first applies depthwise convolution on each channel, and then uses pointwise convolution to integrate and adjust information in the channel dimension. In depthwise separable convolution, pointwise convolution is used to integrate and adjust information in the channel dimension. Pointwise convolution is a convolution operation that only considers a single pixel position, and it uses a 1x1 convolution kernel to perform convolution operations on each channel of the input. Specifically, pointwise convolution generates pixel values ​​for each channel of the output tensor by multiplying and summing the pixel values ​​of each channel of the input tensor with the corresponding convolution kernel weights element by element. This operation can be regarded as a fully connected linear transformation of each channel. Pointwise convolution has no cross-channel connections, it only performs independent operations within the channel, so it can be regarded as integrating and adjusting information in the channel dimension. Depthwise convolution is another part of depthwise separable convolution; in the depthwise convolution stage, for each channel of the input tensor, a depthwise convolution kernel is used to perform a convolution operation on that channel. The kernel size of the depthwise convolution is usually smaller than the number of input channels, so it only performs convolution operations within each channel without considering the relationship between channels. This method effectively reduces the number of parameters because the depthwise convolution is only performed on each channel, not on the entire feature map. By combining depthwise convolution with pointwise convolution, depthwise separable convolution achieves information integration and adjustment in the channel dimension. First, depthwise convolution is applied to each channel, and then pointwise convolution is used to integrate and adjust the information of the output of each channel. Pointwise convolution is equivalent to a fully connected operation between channels, so that channels can interact and integrate, providing richer feature representation. This structure effectively reduces the number of parameters because the depthwise convolution is only performed on each channel, not on the entire feature map. At the same time, pointwise convolution further improves the expressiveness of features and provides richer feature representation through interaction and integration between channels.

[0085] After the depthwise separable convolution, the decomposed sub-channel groups are fused to obtain the fused feature map A2. The fusion method can be a weighted summation at the channel level, where the weights of different sub-channel groups can be learned or manually set according to specific tasks and data.

[0086] After fusion, the fused feature map A2 is residually connected to the feature map to obtain feature information A2X; the original feature information is retained to avoid information loss. Residual connection helps to alleviate the gradient vanishing problem and promote network training and feature transfer. The residual connection is a cross-layer connection method introduced in the neural network. It obtains a residual connection by directly adding the input signal of the previous layer to the output of the subsequent layer. The input of each layer undergoes a series of transformations, such as convolution, full connection, etc., to obtain the output of the layer. The residual connection introduces a jump connection in this process, which directly adds the input of the previous layer to the output of the subsequent layer. Specifically, the residual connection can be expressed as:

[0087] Output = Input + F (Input)

[0088] Among them, input represents the input signal of the previous layer, and F(input) represents the mapping function from input to output, for example, the transformation obtained by operations such as convolution and full connection. Through residual connection, the network can learn the residual part, that is, the difference between the input and the target, instead of relying entirely on the network's own transformation ability.

[0089] Through residual connections, the network can learn the residual part more easily, and the gradient can more easily maintain a stable size in cross-layer propagation. This is because during the back-propagation process, the gradient can be directly passed through the residual connection without being affected by the previous layer transformation. This helps solve the gradient disappearance and gradient explosion problems, making the network training more stable and efficient.

[0090] Through the above steps, each Fused-MBConv convolution block implements the operations of decomposition, depthwise separable convolution, pointwise convolution, fusion and residual connection of the segmented image, providing an effective feature extraction and information fusion method, and enhancing the performance of network feature extraction. Through channel decomposition, depthwise separable convolution and pointwise convolution, the network can better capture features of different scales and abstraction levels. The fusion operation combines the feature information of different sub-channel groups, enriching the diversity of feature representation, while the residual connection retains the original feature information.

[0091] S23, the feature information A2X is further subjected to feature extraction and information fusion through MBConv convolution to obtain feature information A3X; the number of MBConv convolutions can be adaptively selected according to the specific situation. In the present application, the feature information A2X is best subjected to three MBConv convolutions. Each MBConv convolution block generates a new feature map A3 through depth-separable convolution, and a nonlinear activation function is used to activate the new feature map A3. The nonlinear activation function introduces nonlinear transformations, allowing the network to learn nonlinear relationships and more complex feature representations. Common nonlinear activation functions include ReLU (Rectified Linear Unit), which applies the following formula to each element in the new feature map A3: f(x) = max(0, x). The ReLU function sets all negative values ​​to zero and retains positive values; this activation function has simple calculations and good convergence.

[0092] A 1x1 point-by-point convolution operation is applied to adjust the number of channels to obtain the output feature map A4. The 1x1 point-by-point convolution achieves linear combination and adjustment between channels by applying a 1x1 convolution kernel on each channel; it can increase or decrease the number of channels and adjust the relationship between channels of the feature map to adapt to subsequent feature fusion and learning.

[0093] S24, the output feature map A4 of the point-by-point convolution is added to the input feature map A1 to obtain a final feature map A5. Further, the feature information A3X is globally averaged pooled to reduce the spatial dimension of the final feature map A5 to 1x1.

[0094] S25, the high-dimensional features are mapped to the probability distribution of the object target category through the fully connected layer, and an object feature vector with a dimension of 512 can be obtained. The high-dimensional features refer to the feature information extracted by the feature extraction network, which has a higher dimension; for example, the semantic information of the height, width, color and shape of the object is mapped to the probability distribution of the target category; the probability distribution of the target category refers to the probability distribution of mapping the object feature vector to each category, which represents the network's confidence or possibility for each category. For example, assuming that the object target categories include cats, dogs and cars, the output is a vector of size 3, in which each element represents the probability of the corresponding category; if the output vector is [0.1, 0.7, 0.2], it means that the model has a probability of 0.1 for cats, 0.7 for dogs, and 0.2 for cars, that is, the object feature vector of the specified dimension can be obtained. In the implementation method, the dimension is specified as 512 after considering the balance between computing power and accuracy, so as to balance the accuracy and matching speed when matching high-dimensional features of objects.

[0095] In this application, the white area in the binary image obtained by segmentation is the object target, and the black area is the background. The segmented image is contour detected and the maximum circumscribed rectangle is extracted to obtain the object target image T0, T1...Tn .

[0096] After the feature extraction step, each object target will have a corresponding feature vector. These feature vectors can be expressed as:

[0097] Eigenvector 1: F1 = [f 1-1 ,f 1-2 ,f 1-3 ,...,f 1-512 ]

[0098] Eigenvector 2: F2 = [f 2-1 ,f 2-2 ,f 2-3 ,...,f 2-512 ]

[0099] ·...

[0100] Eigenvector M:F M =[f M-1 ,f M-2 ,f M-3 ,...,f M-512 ]

[0101] Here M is the number of object categories, and each eigenvector has 512 eigenvalues.

[0102] According to the above method, the segmentation mask image is obtained, and the segmentation image is input into the feature extraction network, and the object information is converted into a 512-dimensional feature vector. The feature extraction network uses the lightweight network EfficientNetV2-s, and the network structure is as follows:

[0103]

[0104]

[0105] The EfficientNetV2-s network uses perceptual neural architecture search technology and combines it with the composite model expansion method to optimize model accuracy, training speed, and parameter size, and achieves faster training by using MBConv and Fused-MBConv modules.

[0106] Several common semantic segmentation models Res-UNet++ and U2-Net are used to compare the segmentation of the original image with the semantic segmentation model ToolNet proposed in this application. The test results are shown in the following table:

[0107]

[0108] As can be seen from the table, the semantic segmentation model ToolNet in this application is better than U 2-Net has been improved in both IoU and precision, and the average segmentation time has been reduced from 1.27s to 0.56s, and the segmentation efficiency has been improved by 54.47%. It has been further verified that the semantic segmentation model ToolNet uses the ToolRSU module to reduce spatial redundancy and computational complexity, thereby reducing the time of segmentation processing. By introducing a dual attention mechanism to enhance the characteristics of high-frequency and low-frequency components, feature enhancement is performed to improve segmentation accuracy.

[0109] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. An image segmentation and feature extraction method, characterized in that: The following steps are involved: S1, use the semantic segmentation model ToolNet to segment the original image to obtain a mask map, and segment the mask map to obtain a segmented image; S2, inputs the segmented image into the feature extraction network, converts it into a feature vector with rich representation capabilities, and extracts the feature vector.

2. The image segmentation and feature extraction method according to claim 1, characterized in that: A dual attention mechanism is adopted to improve the segmentation accuracy of the semantic segmentation model ToolNet. The dual attention mechanism is two attention mechanisms including temporal attention and spatial attention mechanism.

3. The image segmentation and feature extraction method according to claim 1 or 2, characterized in that: The ToolRSU module is used to improve the segmentation accuracy of the semantic segmentation model ToolNet; the ToolRSU module uses octave convolution to use the feature information of local context and global context to improve the segmentation effect.

4. The image segmentation and feature extraction method according to claim 1, characterized in that: The semantic segmentation model ToolNet includes an encoding layer, a decoding layer and a fully connected layer, and each layer includes at least one U-shaped residual block.

5. The image segmentation and feature extraction method according to claim 4, characterized in that: The encoding layer has a six-layer structure, and the decoding layer has a five-layer structure.

6. The image segmentation and feature extraction method according to claim 3, characterized in that: Input the feature map X and the convolution kernel W into the octave convolution layer, and divide the input feature map X and the convolution kernel W into high-frequency components and low-frequency components. The low-frequency component refers to the feature map obtained after Gaussian filtering, and the high-frequency component refers to the original feature map without Gaussian filtering; L and X H Respectively represent the low-frequency component and high-frequency component of the feature map, W L and W H Represent the convolution kernels for low-frequency components and high-frequency components respectively.

7. The image segmentation and feature extraction method according to claim 6, characterized in that: When extracting features of low-frequency components and high-frequency components in octave convolution, effective communication is performed between the feature representations of low-frequency components and high-frequency components.

8. The image segmentation and feature extraction method according to claim 6, characterized in that: When the feature information is updated from low frequency to high frequency, the low frequency component needs to be upsampled. The calculation formula is: Y H =f(X H ;W H→H )+Upsampling(f(X L ;W L→H ),k) Where f(X;W) represents convolution with kernel W parameters, Upsampling(f(),k) represents upsampling, and W L→H It represents the process of updating from low frequency to high frequency, k represents the stride, and the nearest neighbor interpolation with stride k=2.

9. The image segmentation and feature extraction method according to claim 6, characterized in that: When feature information is updated from high frequency to low frequency, average pooling is required to downsample the high frequency components. The calculation formula is: Y L =f(X L ;W L→L )+f(pool(X H ,k);W H→L ) Where pool(X,k) represents the average pooling operation to achieve downsampling, W H→L It represents the process of updating information from high frequency to low frequency, k represents the stride; stride k=2.

10. The image segmentation and feature extraction method according to claim 6, characterized in that: The feature map size of the low-frequency component is set to half of the feature map size of the high-frequency component.

11. The image segmentation and feature extraction method according to claim 1, characterized in that: The feature extraction network uses the lightweight network EfficientNetV2-s.

12. The image segmentation and feature extraction method according to claim 2, characterized in that S2 include: S21, input the segmented image into the feature extraction network, and perform shallow feature extraction on the segmented image after convolution with a convolution kernel to obtain feature information A1X; S22, feature information A1X is further subjected to feature extraction and information fusion through Fused-MBConv convolution to obtain feature information A2X; S23, feature information A2X is further subjected to feature extraction and information fusion through MBConv convolution to obtain feature information A3X; S24, feature information A3X is globally averaged pooled to reduce the spatial dimension of the final feature map to 1x1; S25 maps the feature information A3X to the probability distribution of the object target category through a fully connected layer to obtain an object feature vector with a dimension of 512.

13. The image segmentation and feature extraction method according to claim 12, characterized in that: In S21, the size of the convolution kernel is 3×3.

14. The image segmentation and feature extraction method according to claim 12, characterized in that: Each Fused-MBConv convolution block contains five steps: decomposition of the input feature map, depth-wise separable convolution, point-wise convolution, fusion, and residual connection.

15. The image segmentation and feature extraction method according to claim 3, characterized in that: Low frequency characteristics Upsample from a layer in the encoding stage to the same scale as the high-frequency component It is used for channel splicing, and then the feature map is converted into a spatial attention weight map through a 1*1 convolution operation; and Represent the feature maps of high-frequency components and low-frequency components of the i-th layer respectively.

16. The image segmentation and feature extraction method according to claim 15, characterized in that: For the high-frequency component of the feature map, the generated attention weight map is first normalized by the sigmoid function, and then directly multiplied with the feature map of the high-frequency component to obtain the high-frequency feature map Where δ() represents the sigmoid function, Conv 1X1 () represents the 1*1 convolution after channel splicing and fusion; It means that the generated spatial attention map is multiplied by the corresponding point of each high-frequency feature map.

17. The image segmentation and feature extraction method according to claim 15, characterized in that: For the low-frequency component of the feature map, the generated attention weight map is first downsampled to the same scale as the low-frequency component, then normalized using the sigmoid function, and finally multiplied with the low-frequency component to obtain the low-frequency feature map. Where Sub() represents the downsampling function; It means that the generated spatial attention map is multiplied by the corresponding point of each low-frequency feature map.

18. The image segmentation and feature extraction method according to claim 11, characterized in that: The input size of the feature extraction network is 224×224×3.

Images